[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:30:33 dist-test-slave-khd3 krb5kdc[6283](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:30:33 dist-test-slave-khd3 krb5kdc[6283](info): set up 2 sockets
Sep 26 16:30:33 dist-test-slave-khd3 krb5kdc[6283](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260926 16:30:34.983940 6276 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.001s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Sep 26 16:30:35 dist-test-slave-khd3 krb5kdc[6283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:30:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:30:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab.
Entry for principal kudu/127.6.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab.
I20260926 16:30:35.165969 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:40861
--webserver_interface=127.6.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:32915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.33.62:40861
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:30:35.489665 6296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:30:35.490041 6296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:30:35.490199 6296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:30:35.497442 6296 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:30:35.497591 6296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:30:35.497699 6296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:30:35.497795 6296 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:30:35.497880 6296 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:30:35.507875 6296 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:32915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.33.62:40861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:40861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:30:35.509282 6296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:30:35.510610 6296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:30:35.522665 6301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:30:35.522635 6302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:30:35.523262 6304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:30:35.524036 6296 server_base.cc:1034] running on GCE node
I20260926 16:30:35.524976 6296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:30:35.526518 6296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:30:35.527776 6296 hybrid_clock.cc:648] HybridClock initialized: now 1790440235527723 us; error 46 us; skew 500 ppm
I20260926 16:30:35.528169 6296 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:30:35 dist-test-slave-khd3 krb5kdc[6283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440235, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:30:35.531821 6296 init.cc:377] Logged in from keytab as kudu/127.6.33.62@KRBTEST.COM (short username kudu)
I20260926 16:30:35.533520 6296 webserver.cc:460] Webserver started at http://127.6.33.62:34121/ using document root <none> and password file <none>
I20260926 16:30:35.534251 6296 fs_manager.cc:359] Metadata directory not provided
I20260926 16:30:35.534407 6296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:30:35.534724 6296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:30:35.537916 6296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/minicluster-data/master-0/data/instance:
uuid: "8f5d6ebf80b24641ad3bf7ee8151d723"
format_stamp: "Formatted at 2026-09-26 16:30:35 on dist-test-slave-khd3"
server_key: "b9850660dcb00451abd4294de61e511b"
server_key_iv: "f1f179b9db9f7ca75669572fe290d359"
server_key_version: "encryptionkey@0"
I20260926 16:30:35.538697 6296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/minicluster-data/master-0/wal/instance:
uuid: "8f5d6ebf80b24641ad3bf7ee8151d723"
format_stamp: "Formatted at 2026-09-26 16:30:35 on dist-test-slave-khd3"
server_key: "b9850660dcb00451abd4294de61e511b"
server_key_iv: "f1f179b9db9f7ca75669572fe290d359"
server_key_version: "encryptionkey@0"
I20260926 16:30:35.544304 6296 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 16:30:35.548135 6311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:30:35.549712 6296 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:30:35.549943 6296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/minicluster-data/master-0/wal
uuid: "8f5d6ebf80b24641ad3bf7ee8151d723"
format_stamp: "Formatted at 2026-09-26 16:30:35 on dist-test-slave-khd3"
server_key: "b9850660dcb00451abd4294de61e511b"
server_key_iv: "f1f179b9db9f7ca75669572fe290d359"
server_key_version: "encryptionkey@0"
I20260926 16:30:35.550190 6296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:30:35.573788 6296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:30:35.578850 6296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:30:35.579180 6296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:30:35.598646 6296 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.62:40861
I20260926 16:30:35.598690 6362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.62:40861 every 8 connection(s)
I20260926 16:30:35.600071 6296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
I20260926 16:30:35.603452 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 6296
I20260926 16:30:35.603765 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/minicluster-data/master-0/wal/instance
I20260926 16:30:35.604086 6363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:30:35.605791 6276 external_mini_cluster.cc:1428] Setting key 93af2c4af69a2e7b81fe0367cc347b31
I20260926 16:30:35.614853 6363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f5d6ebf80b24641ad3bf7ee8151d723: Bootstrap starting.
I20260926 16:30:35.617933 6363 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f5d6ebf80b24641ad3bf7ee8151d723: Neither blocks nor log segments found. Creating new log.
I20260926 16:30:35.619027 6363 log.cc:824] T 00000000000000000000000000000000 P 8f5d6ebf80b24641ad3bf7ee8151d723: Log is configured to *not* fsync() on all Append() calls
I20260926 16:30:35.622424 6363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f5d6ebf80b24641ad3bf7ee8151d723: No bootstrap required, opened a new log
I20260926 16:30:35.633499 6363 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f5d6ebf80b24641ad3bf7ee8151d723 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f5d6ebf80b24641ad3bf7ee8151d723" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 40861 } }
I20260926 16:30:35.633996 6363 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f5d6ebf80b24641ad3bf7ee8151d723 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:30:35.634110 6363 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f5d6ebf80b24641ad3bf7ee8151d723 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f5d6ebf80b24641ad3bf7ee8151d723, State: Initialized, Role: FOLLOWER
I20260926 16:30:35.634819 6363 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f5d6ebf80b24641ad3bf7ee8151d723 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f5d6ebf80b24641ad3bf7ee8151d723" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 40861 } }
I20260926 16:30:35.635113 6363 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f5d6ebf80b24641ad3bf7ee8151d723 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:30:35.635258 6363 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f5d6ebf80b24641ad3bf7ee8151d723 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:30:35.635489 6363 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f5d6ebf80b24641ad3bf7ee8151d723 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:30:35.638757 6363 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f5d6ebf80b24641ad3bf7ee8151d723 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f5d6ebf80b24641ad3bf7ee8151d723" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 40861 } }
I20260926 16:30:35.639489 6363 leader_election.cc:302] T 00000000000000000000000000000000 P 8f5d6ebf80b24641ad3bf7ee8151d723 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f5d6ebf80b24641ad3bf7ee8151d723; no voters:
I20260926 16:30:35.640254 6363 leader_election.cc:288] T 00000000000000000000000000000000 P 8f5d6ebf80b24641ad3bf7ee8151d723 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:30:35.640368 6368 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f5d6ebf80b24641ad3bf7ee8151d723 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:30:35.642359 6368 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f5d6ebf80b24641ad3bf7ee8151d723 [term 1 LEADER]: Becoming Leader. State: Replica: 8f5d6ebf80b24641ad3bf7ee8151d723, State: Running, Role: LEADER
Sep 26 16:30:35 dist-test-slave-khd3 krb5kdc[6283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:30:35.642875 6363 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f5d6ebf80b24641ad3bf7ee8151d723 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:30:35.643036 6368 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f5d6ebf80b24641ad3bf7ee8151d723 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f5d6ebf80b24641ad3bf7ee8151d723" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 40861 } }
I20260926 16:30:35.649247 6369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f5d6ebf80b24641ad3bf7ee8151d723 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f5d6ebf80b24641ad3bf7ee8151d723" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f5d6ebf80b24641ad3bf7ee8151d723" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 40861 } } }
I20260926 16:30:35.649655 6369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f5d6ebf80b24641ad3bf7ee8151d723 [sys.catalog]: This master's current role is: LEADER
I20260926 16:30:35.653631 6370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f5d6ebf80b24641ad3bf7ee8151d723 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f5d6ebf80b24641ad3bf7ee8151d723. Latest consensus state: current_term: 1 leader_uuid: "8f5d6ebf80b24641ad3bf7ee8151d723" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f5d6ebf80b24641ad3bf7ee8151d723" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 40861 } } }
I20260926 16:30:35.654183 6370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f5d6ebf80b24641ad3bf7ee8151d723 [sys.catalog]: This master's current role is: LEADER
I20260926 16:30:35.656015 6374 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:30:35.660662 6365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:35.609780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52786 (local address 127.6.33.62:40861)
0926 16:30:35.610709 (+ 929us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:35.610724 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:35.614425 (+ 3701us) server_negotiation.cc:407] Connection header received
0926 16:30:35.624095 (+ 9670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:35.624128 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:35.624709 (+ 581us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:35.625253 (+ 544us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:30:35.630352 (+ 5099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:35.632698 (+ 2346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:35.635462 (+ 2764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:35.636387 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:35.636551 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:35.644802 (+ 8251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:30:35.644818 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:30:35.644847 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:30:35.644927 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:30:35.653234 (+ 8307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:35.654698 (+ 1464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:35.654706 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:35.654716 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:35.654988 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:35.655635 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:35.655644 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:35.655649 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:35.656469 (+ 820us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:30:35.656733 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:35.658226 (+ 1493us) server_negotiation.cc:299] Negotiation successful
0926 16:30:35.658726 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":671,"thread_start_us":396,"threads_started":1}
I20260926 16:30:35.661808 6374 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:30:35.673135 6374 catalog_manager.cc:1347] Generated new cluster ID: 30ffefbb85c94e6095275f9934ac61b2
I20260926 16:30:35.673278 6374 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:30:35.698158 6374 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:30:35.699605 6374 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:30:35.708133 6374 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f5d6ebf80b24641ad3bf7ee8151d723: Generated new TSK 0
I20260926 16:30:35.709363 6374 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.33.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab.
Entry for principal kudu/127.6.33.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab.
I20260926 16:30:35.761284 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.1:0
--local_ip_for_outbound_sockets=127.6.33.1
--webserver_interface=127.6.33.1
--webserver_port=0
--tserver_master_addrs=127.6.33.62:40861
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab
--principal=kudu/127.6.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:32915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:30:36.108064 6389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:30:36.108347 6389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:30:36.108446 6389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:30:36.115708 6389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:30:36.115840 6389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:30:36.115962 6389 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.1
I20260926 16:30:36.126232 6389 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:32915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab
--principal=kudu/127.6.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.33.1
--webserver_port=0
--tserver_master_addrs=127.6.33.62:40861
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.1
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:30:36.127578 6389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:30:36.128976 6389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:30:36.142799 6394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:30:36.142797 6395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:30:36.142668 6397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:30:36.143817 6389 server_base.cc:1034] running on GCE node
I20260926 16:30:36.144550 6389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:30:36.145527 6389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:30:36.146723 6389 hybrid_clock.cc:648] HybridClock initialized: now 1790440236146656 us; error 68 us; skew 500 ppm
I20260926 16:30:36.147074 6389 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:30:36 dist-test-slave-khd3 krb5kdc[6283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440236, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:30:36.150583 6389 init.cc:377] Logged in from keytab as kudu/127.6.33.1@KRBTEST.COM (short username kudu)
I20260926 16:30:36.152202 6389 webserver.cc:460] Webserver started at http://127.6.33.1:32999/ using document root <none> and password file <none>
I20260926 16:30:36.153017 6389 fs_manager.cc:359] Metadata directory not provided
I20260926 16:30:36.153159 6389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:30:36.153481 6389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:30:36.156744 6389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/minicluster-data/ts-0/data/instance:
uuid: "805e9c69ca4047f38cd73956a60691fe"
format_stamp: "Formatted at 2026-09-26 16:30:36 on dist-test-slave-khd3"
server_key: "01ef45a6f948dfaa537212c23b82eb8e"
server_key_iv: "2233ee14a0b4719c3b8eebe7ad4c4782"
server_key_version: "encryptionkey@0"
I20260926 16:30:36.157728 6389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/minicluster-data/ts-0/wal/instance:
uuid: "805e9c69ca4047f38cd73956a60691fe"
format_stamp: "Formatted at 2026-09-26 16:30:36 on dist-test-slave-khd3"
server_key: "01ef45a6f948dfaa537212c23b82eb8e"
server_key_iv: "2233ee14a0b4719c3b8eebe7ad4c4782"
server_key_version: "encryptionkey@0"
I20260926 16:30:36.163457 6389 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260926 16:30:36.167750 6404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:30:36.169457 6389 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260926 16:30:36.169636 6389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/minicluster-data/ts-0/wal
uuid: "805e9c69ca4047f38cd73956a60691fe"
format_stamp: "Formatted at 2026-09-26 16:30:36 on dist-test-slave-khd3"
server_key: "01ef45a6f948dfaa537212c23b82eb8e"
server_key_iv: "2233ee14a0b4719c3b8eebe7ad4c4782"
server_key_version: "encryptionkey@0"
I20260926 16:30:36.169845 6389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:30:36.207121 6389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:30:36.211485 6389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:30:36.211778 6389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:30:36.212926 6389 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:30:36.214275 6389 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:30:36.214347 6389 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:30:36.214521 6389 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:30:36.214583 6389 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:30:36.247398 6389 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.1:43145
I20260926 16:30:36.247486 6516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.1:43145 every 8 connection(s)
I20260926 16:30:36.248718 6389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
I20260926 16:30:36.249106 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 6389
I20260926 16:30:36.249356 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/minicluster-data/ts-0/wal/instance
I20260926 16:30:36.250186 6276 external_mini_cluster.cc:1428] Setting key 2bc56f8cd362f580795838e811a8c1a4
Sep 26 16:30:36 dist-test-slave-khd3 krb5kdc[6283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440236, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.1@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.33.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.2@KRBTEST.COM" created.
I20260926 16:30:36.273686 6519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:36.251913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:58627 (local address 127.6.33.62:40861)
0926 16:30:36.252396 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:36.252404 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:36.253737 (+ 1333us) server_negotiation.cc:407] Connection header received
0926 16:30:36.256957 (+ 3220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:36.256963 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:36.257045 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:36.257140 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:30:36.259577 (+ 2437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:36.260307 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:36.262120 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:36.262538 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:36.262683 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:36.266940 (+ 4257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:30:36.266945 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:30:36.266949 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:30:36.266985 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:30:36.269886 (+ 2901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:36.270912 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:36.270917 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:36.270919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:36.271050 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:36.271638 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:36.271642 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:36.271644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:36.271913 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:30:36.272037 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:36.273118 (+ 1081us) server_negotiation.cc:299] Negotiation successful
0926 16:30:36.273348 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":242,"threads_started":1}
I20260926 16:30:36.275012 6520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:36.252632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:40861 (local address 127.6.33.1:58627)
0926 16:30:36.253562 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0926 16:30:36.253623 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 16:30:36.256408 (+ 2785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:30:36.257508 (+ 1100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:30:36.257517 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:30:36.258404 (+ 887us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:30:36.259372 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:36.259393 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:36.260465 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:36.260473 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:36.261966 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:36.261979 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:36.262784 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:36.262793 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:36.263070 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:36.264011 (+ 941us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:30:36.264044 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:30:36.266630 (+ 2586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:30:36.270035 (+ 3405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:30:36.270048 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:30:36.270071 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:30:36.270756 (+ 685us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:30:36.271171 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:30:36.271179 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:30:36.271186 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:30:36.271501 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:30:36.272052 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:30:36.272065 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:30:36.272604 (+ 539us) client_negotiation.cc:769] Sending connection context
0926 16:30:36.272949 (+ 345us) client_negotiation.cc:240] Negotiation successful
0926 16:30:36.273606 (+ 657us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":313,"threads_started":1}
I20260926 16:30:36.276693 6517 heartbeater.cc:347] Connected to a master server at 127.6.33.62:40861
I20260926 16:30:36.277076 6517 heartbeater.cc:464] Registering TS with master...
I20260926 16:30:36.277858 6517 heartbeater.cc:511] Master 127.6.33.62:40861 requested a full tablet report, sending...
I20260926 16:30:36.280331 6328 ts_manager.cc:194] Registered new tserver with Master: 805e9c69ca4047f38cd73956a60691fe (127.6.33.1:43145)
I20260926 16:30:36.281980 6328 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.33.1@KRBTEST.COM'} at 127.6.33.1:58627
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab.
Entry for principal kudu/127.6.33.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab.
I20260926 16:30:36.293187 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.2:0
--local_ip_for_outbound_sockets=127.6.33.2
--webserver_interface=127.6.33.2
--webserver_port=0
--tserver_master_addrs=127.6.33.62:40861
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab
--principal=kudu/127.6.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:32915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:30:36.624725 6523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:30:36.625119 6523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:30:36.625279 6523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:30:36.632542 6523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:30:36.632648 6523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:30:36.632889 6523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.2
I20260926 16:30:36.642570 6523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:32915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab
--principal=kudu/127.6.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.33.2
--webserver_port=0
--tserver_master_addrs=127.6.33.62:40861
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.2
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:30:36.643918 6523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:30:36.645282 6523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:30:36.658270 6531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:30:36.658347 6529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:30:36.658321 6528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:30:36.659713 6523 server_base.cc:1034] running on GCE node
I20260926 16:30:36.660383 6523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:30:36.661466 6523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:30:36.662722 6523 hybrid_clock.cc:648] HybridClock initialized: now 1790440236662651 us; error 58 us; skew 500 ppm
I20260926 16:30:36.663136 6523 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:30:36 dist-test-slave-khd3 krb5kdc[6283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440236, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:30:36.666507 6523 init.cc:377] Logged in from keytab as kudu/127.6.33.2@KRBTEST.COM (short username kudu)
I20260926 16:30:36.668084 6523 webserver.cc:460] Webserver started at http://127.6.33.2:39549/ using document root <none> and password file <none>
I20260926 16:30:36.668859 6523 fs_manager.cc:359] Metadata directory not provided
I20260926 16:30:36.668972 6523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:30:36.669287 6523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:30:36.672312 6523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/minicluster-data/ts-1/data/instance:
uuid: "1ea11b4034ff41b8a42862ff126cc4a2"
format_stamp: "Formatted at 2026-09-26 16:30:36 on dist-test-slave-khd3"
server_key: "50f09a37a69888fa1c940e023ce9a946"
server_key_iv: "d19816f704753cca219a1ad0dbcde4d3"
server_key_version: "encryptionkey@0"
I20260926 16:30:36.673276 6523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/minicluster-data/ts-1/wal/instance:
uuid: "1ea11b4034ff41b8a42862ff126cc4a2"
format_stamp: "Formatted at 2026-09-26 16:30:36 on dist-test-slave-khd3"
server_key: "50f09a37a69888fa1c940e023ce9a946"
server_key_iv: "d19816f704753cca219a1ad0dbcde4d3"
server_key_version: "encryptionkey@0"
I20260926 16:30:36.678831 6523 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 16:30:36.682528 6538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:30:36.683997 6523 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 16:30:36.684278 6523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/minicluster-data/ts-1/wal
uuid: "1ea11b4034ff41b8a42862ff126cc4a2"
format_stamp: "Formatted at 2026-09-26 16:30:36 on dist-test-slave-khd3"
server_key: "50f09a37a69888fa1c940e023ce9a946"
server_key_iv: "d19816f704753cca219a1ad0dbcde4d3"
server_key_version: "encryptionkey@0"
I20260926 16:30:36.684517 6523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:30:36.703029 6523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:30:36.707880 6523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:30:36.708214 6523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:30:36.709381 6523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:30:36.710914 6523 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:30:36.711092 6523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:30:36.711261 6523 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:30:36.711342 6523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:30:36.745298 6523 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.2:36565
I20260926 16:30:36.745388 6650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.2:36565 every 8 connection(s)
I20260926 16:30:36.746667 6523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
I20260926 16:30:36.750487 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 6523
I20260926 16:30:36.750686 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/minicluster-data/ts-1/wal/instance
I20260926 16:30:36.751235 6276 external_mini_cluster.cc:1428] Setting key 7adab01d8cb2a2d036be242816c3836c
Sep 26 16:30:36 dist-test-slave-khd3 krb5kdc[6283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440236, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.2@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.33.3@KRBTEST.COM; defaulting to no policy
I20260926 16:30:36.769047 6519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:36.749297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:50449 (local address 127.6.33.62:40861)
0926 16:30:36.749474 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:36.749479 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:36.750748 (+ 1269us) server_negotiation.cc:407] Connection header received
0926 16:30:36.753046 (+ 2298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:36.753051 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:36.753142 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:36.753249 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:30:36.755074 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:36.756224 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:36.758262 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:36.758827 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:36.758958 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:36.763204 (+ 4246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:30:36.763208 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:30:36.763212 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:30:36.763242 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:30:36.765946 (+ 2704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:36.766690 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:36.766697 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:36.766706 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:36.766826 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:36.767283 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:36.767288 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:36.767291 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:36.767554 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:30:36.767718 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:36.768444 (+ 726us) server_negotiation.cc:299] Negotiation successful
0926 16:30:36.768660 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.3@KRBTEST.COM" created.
I20260926 16:30:36.770162 6653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:36.749705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:40861 (local address 127.6.33.2:50449)
0926 16:30:36.750561 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0926 16:30:36.750613 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 16:30:36.752704 (+ 2091us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:30:36.753555 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:30:36.753564 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:30:36.754190 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:30:36.754869 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:36.754887 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:36.756387 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:36.756391 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:36.758119 (+ 1728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:36.758135 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:36.759075 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:36.759085 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:36.759438 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:36.760169 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:30:36.760207 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:30:36.762970 (+ 2763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:30:36.766093 (+ 3123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:30:36.766103 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:30:36.766121 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:30:36.766560 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:30:36.766941 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:30:36.766947 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:30:36.766950 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:30:36.767164 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:30:36.767686 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:30:36.767697 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:30:36.768083 (+ 386us) client_negotiation.cc:769] Sending connection context
0926 16:30:36.768355 (+ 272us) client_negotiation.cc:240] Negotiation successful
0926 16:30:36.769004 (+ 649us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":316,"threads_started":1}
I20260926 16:30:36.771809 6651 heartbeater.cc:347] Connected to a master server at 127.6.33.62:40861
I20260926 16:30:36.772172 6651 heartbeater.cc:464] Registering TS with master...
I20260926 16:30:36.773141 6651 heartbeater.cc:511] Master 127.6.33.62:40861 requested a full tablet report, sending...
I20260926 16:30:36.774967 6328 ts_manager.cc:194] Registered new tserver with Master: 1ea11b4034ff41b8a42862ff126cc4a2 (127.6.33.2:36565)
I20260926 16:30:36.775988 6328 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.33.2@KRBTEST.COM'} at 127.6.33.2:50449
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab.
Entry for principal kudu/127.6.33.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab.
I20260926 16:30:36.792346 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.3:0
--local_ip_for_outbound_sockets=127.6.33.3
--webserver_interface=127.6.33.3
--webserver_port=0
--tserver_master_addrs=127.6.33.62:40861
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab
--principal=kudu/127.6.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:32915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:30:37.128044 6656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:30:37.128341 6656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:30:37.128439 6656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:30:37.135975 6656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:30:37.136096 6656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:30:37.136212 6656 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.3
I20260926 16:30:37.146562 6656 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:32915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab
--principal=kudu/127.6.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.33.3
--webserver_port=0
--tserver_master_addrs=127.6.33.62:40861
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.3
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:30:37.147984 6656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:30:37.149710 6656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:30:37.164083 6664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:30:37.164047 6662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:30:37.164970 6661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:30:37.166523 6656 server_base.cc:1034] running on GCE node
I20260926 16:30:37.167200 6656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:30:37.168196 6656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:30:37.169425 6656 hybrid_clock.cc:648] HybridClock initialized: now 1790440237169355 us; error 73 us; skew 500 ppm
I20260926 16:30:37.169760 6656 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:30:37 dist-test-slave-khd3 krb5kdc[6283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440237, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:30:37.193291 6656 init.cc:377] Logged in from keytab as kudu/127.6.33.3@KRBTEST.COM (short username kudu)
I20260926 16:30:37.195061 6656 webserver.cc:460] Webserver started at http://127.6.33.3:35899/ using document root <none> and password file <none>
I20260926 16:30:37.195839 6656 fs_manager.cc:359] Metadata directory not provided
I20260926 16:30:37.195990 6656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:30:37.196314 6656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:30:37.199599 6656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/minicluster-data/ts-2/data/instance:
uuid: "f33315d0661c451f8adee2c2816e8f24"
format_stamp: "Formatted at 2026-09-26 16:30:37 on dist-test-slave-khd3"
server_key: "b089a12aea19493388a2bdbbf12a0676"
server_key_iv: "e9d98981b1e9c9f68c9dbfb5e08719c3"
server_key_version: "encryptionkey@0"
I20260926 16:30:37.200454 6656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/minicluster-data/ts-2/wal/instance:
uuid: "f33315d0661c451f8adee2c2816e8f24"
format_stamp: "Formatted at 2026-09-26 16:30:37 on dist-test-slave-khd3"
server_key: "b089a12aea19493388a2bdbbf12a0676"
server_key_iv: "e9d98981b1e9c9f68c9dbfb5e08719c3"
server_key_version: "encryptionkey@0"
I20260926 16:30:37.206051 6656 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 16:30:37.209905 6671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:30:37.211272 6656 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 16:30:37.211448 6656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/minicluster-data/ts-2/wal
uuid: "f33315d0661c451f8adee2c2816e8f24"
format_stamp: "Formatted at 2026-09-26 16:30:37 on dist-test-slave-khd3"
server_key: "b089a12aea19493388a2bdbbf12a0676"
server_key_iv: "e9d98981b1e9c9f68c9dbfb5e08719c3"
server_key_version: "encryptionkey@0"
I20260926 16:30:37.211663 6656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:30:37.231087 6656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:30:37.235816 6656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:30:37.236112 6656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:30:37.237156 6656 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:30:37.238514 6656 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:30:37.238615 6656 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:30:37.238778 6656 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:30:37.238858 6656 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:30:37.273413 6656 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.3:36555
I20260926 16:30:37.273510 6783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.3:36555 every 8 connection(s)
I20260926 16:30:37.274788 6656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
I20260926 16:30:37.280207 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 6656
I20260926 16:30:37.280406 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440233850748-6276-0/minicluster-data/ts-2/wal/instance
I20260926 16:30:37.280857 6276 external_mini_cluster.cc:1428] Setting key 9aa38b00c0336319a2889791db002c5c
I20260926 16:30:37.285249 6517 heartbeater.cc:503] Master 127.6.33.62:40861 was elected leader, sending a full tablet report...
Sep 26 16:30:37 dist-test-slave-khd3 krb5kdc[6283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440237, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.3@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:30:37.297101 6786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:37.277492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.3:35101 (local address 127.6.33.62:40861)
0926 16:30:37.277971 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:37.277976 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:37.278992 (+ 1016us) server_negotiation.cc:407] Connection header received
0926 16:30:37.281086 (+ 2094us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:37.281092 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:37.281181 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:37.281320 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:30:37.283410 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:37.284494 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:37.286641 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:37.287154 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:37.287254 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:37.291026 (+ 3772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:30:37.291031 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:30:37.291035 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:30:37.291065 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:30:37.293695 (+ 2630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:37.294454 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:37.294457 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:37.294460 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:37.294578 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:37.295018 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:37.295022 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:37.295024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:37.295341 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:30:37.295461 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:37.296447 (+ 986us) server_negotiation.cc:299] Negotiation successful
0926 16:30:37.296627 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":242,"threads_started":1}
I20260926 16:30:37.298243 6787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:37.278113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:40861 (local address 127.6.33.3:35101)
0926 16:30:37.278816 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0926 16:30:37.278870 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 16:30:37.280698 (+ 1828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:30:37.281532 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:30:37.281540 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:30:37.282328 (+ 788us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:30:37.283075 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:37.283093 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:37.284649 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:37.284655 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:37.286488 (+ 1833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:37.286500 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:37.287367 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:37.287536 (+ 169us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:37.287766 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:37.288445 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:30:37.288473 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:30:37.290806 (+ 2333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:30:37.293838 (+ 3032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:30:37.293845 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:30:37.293861 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:30:37.294328 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:30:37.294670 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:30:37.294674 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:30:37.294676 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:30:37.294859 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:30:37.295485 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:30:37.295493 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:30:37.295983 (+ 490us) client_negotiation.cc:769] Sending connection context
0926 16:30:37.296306 (+ 323us) client_negotiation.cc:240] Negotiation successful
0926 16:30:37.296959 (+ 653us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":268,"threads_started":1}
I20260926 16:30:37.299671 6784 heartbeater.cc:347] Connected to a master server at 127.6.33.62:40861
I20260926 16:30:37.300024 6784 heartbeater.cc:464] Registering TS with master...
I20260926 16:30:37.300851 6784 heartbeater.cc:511] Master 127.6.33.62:40861 requested a full tablet report, sending...
I20260926 16:30:37.302248 6328 ts_manager.cc:194] Registered new tserver with Master: f33315d0661c451f8adee2c2816e8f24 (127.6.33.3:36555)
I20260926 16:30:37.303119 6328 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.33.3@KRBTEST.COM'} at 127.6.33.3:35101
I20260926 16:30:37.309926 6276 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 16:30:37 dist-test-slave-khd3 krb5kdc[6283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440237, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 16:30:37 dist-test-slave-khd3 krb5kdc[6283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440237, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.33.1@KRBTEST.COM
I20260926 16:30:37.357583 6792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:37.340530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50028 (local address 127.6.33.1:43145)
0926 16:30:37.341214 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:37.341220 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:37.341248 (+ 28us) server_negotiation.cc:407] Connection header received
0926 16:30:37.341370 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:37.341377 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:37.341604 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:37.341737 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:30:37.343191 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:37.344348 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:37.345836 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:37.346456 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:37.346572 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:37.351110 (+ 4538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:30:37.351123 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:30:37.351138 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:30:37.351178 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:30:37.354516 (+ 3338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:37.355413 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:37.355422 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:37.355424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:37.355555 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:37.355984 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:37.355988 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:37.355990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:37.356349 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:30:37.356591 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:37.356989 (+ 398us) server_negotiation.cc:299] Negotiation successful
0926 16:30:37.357189 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":405,"threads_started":1}
W20260926 16:30:37.358853 6432 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50028
Sep 26 16:30:37 dist-test-slave-khd3 krb5kdc[6283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 16:30:37 dist-test-slave-khd3 krb5kdc[6283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.1@KRBTEST.COM
I20260926 16:30:37.404533 6792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:37.390113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50036 (local address 127.6.33.1:43145)
0926 16:30:37.390359 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:37.390369 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:37.390703 (+ 334us) server_negotiation.cc:407] Connection header received
0926 16:30:37.390933 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:37.390937 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:37.391016 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:37.391125 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:30:37.392529 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:37.393314 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:37.394605 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:37.395021 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:37.395160 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:37.399362 (+ 4202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:30:37.399368 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:30:37.399372 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:30:37.399415 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:30:37.402041 (+ 2626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:37.402761 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:37.402764 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:37.402767 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:37.402888 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:37.403402 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:37.403407 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:37.403410 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:37.403664 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:30:37.403845 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:37.404134 (+ 289us) server_negotiation.cc:299] Negotiation successful
0926 16:30:37.404302 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 16:30:37.406332 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 6389
I20260926 16:30:37.420157 6792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:37.407347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50052 (local address 127.6.33.1:43145)
0926 16:30:37.407602 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:37.407609 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:37.407856 (+ 247us) server_negotiation.cc:407] Connection header received
0926 16:30:37.408010 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:37.408014 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:37.408078 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:37.408153 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:30:37.409463 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:37.410182 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:37.411428 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:37.411865 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:37.411968 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:37.414392 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:30:37.414397 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:30:37.414400 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:30:37.414434 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:30:37.417418 (+ 2984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:37.418083 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:37.418087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:37.418090 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:37.418217 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:37.418575 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:37.418579 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:37.418582 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:37.418949 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:30:37.419148 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:37.419482 (+ 334us) server_negotiation.cc:299] Negotiation successful
0926 16:30:37.419675 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260926 16:30:37.420969 6512 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:30:37.505426 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 6389
I20260926 16:30:37.522158 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 6523
Sep 26 16:30:37 dist-test-slave-khd3 krb5kdc[6283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.2@KRBTEST.COM
I20260926 16:30:37.538527 6799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:37.523116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39320 (local address 127.6.33.2:36565)
0926 16:30:37.523647 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:37.523655 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:37.523679 (+ 24us) server_negotiation.cc:407] Connection header received
0926 16:30:37.523743 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:37.523750 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:37.523959 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:37.524134 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:30:37.525355 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:37.526447 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:37.527675 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:37.528217 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:37.528360 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:37.532436 (+ 4076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:30:37.532448 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:30:37.532462 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:30:37.532501 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:30:37.535470 (+ 2969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:37.536147 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:37.536154 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:37.536156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:37.536275 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:37.536726 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:37.536732 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:37.536735 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:37.537216 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:30:37.537536 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:37.537867 (+ 331us) server_negotiation.cc:299] Negotiation successful
0926 16:30:37.538150 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":258,"threads_started":1}
I20260926 16:30:37.539806 6646 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:30:37.630714 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 6523
I20260926 16:30:37.652468 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 6656
Sep 26 16:30:37 dist-test-slave-khd3 krb5kdc[6283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.3@KRBTEST.COM
I20260926 16:30:37.669525 6801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:37.653676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44708 (local address 127.6.33.3:36555)
0926 16:30:37.654411 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:37.654419 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:37.654471 (+ 52us) server_negotiation.cc:407] Connection header received
0926 16:30:37.654532 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:37.654561 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:37.654883 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:37.655023 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:30:37.656413 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:37.657639 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:37.659008 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:37.659575 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:37.659723 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:37.663941 (+ 4218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:30:37.663953 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:30:37.663968 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:30:37.664008 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:30:37.666950 (+ 2942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:37.667649 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:37.667656 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:37.667659 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:37.667780 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:37.668091 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:37.668094 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:37.668096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:37.668431 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:30:37.668648 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:37.668942 (+ 294us) server_negotiation.cc:299] Negotiation successful
0926 16:30:37.669140 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":434,"threads_started":1}
I20260926 16:30:37.670881 6779 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:30:37.754989 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 6656
I20260926 16:30:37.772050 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 6296
I20260926 16:30:37.773468 6358 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:30:37.878677 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 6296
2026-09-26T16:30:37Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3995 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:30:37 dist-test-slave-khd3 krb5kdc[6806](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:30:37 dist-test-slave-khd3 krb5kdc[6806](info): set up 2 sockets
Sep 26 16:30:37 dist-test-slave-khd3 krb5kdc[6806](info): commencing operation
krb5kdc: starting...
W20260926 16:30:38.977850 6276 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.002s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Sep 26 16:30:39 dist-test-slave-khd3 krb5kdc[6806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:30:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:30:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab.
Entry for principal kudu/127.6.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab.
I20260926 16:30:39.161074 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:35023
--webserver_interface=127.6.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:36525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.33.62:35023
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:30:39.490407 6819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:30:39.490705 6819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:30:39.490880 6819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:30:39.498221 6819 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:30:39.498337 6819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:30:39.498427 6819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:30:39.498517 6819 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:30:39.498598 6819 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:30:39.508566 6819 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:36525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.33.62:35023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:35023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:30:39.510046 6819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:30:39.511480 6819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:30:39.523845 6824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:30:39.523846 6827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:30:39.523904 6825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:30:39.524971 6819 server_base.cc:1034] running on GCE node
I20260926 16:30:39.525851 6819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:30:39.527519 6819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:30:39.528891 6819 hybrid_clock.cc:648] HybridClock initialized: now 1790440239528735 us; error 50 us; skew 500 ppm
I20260926 16:30:39.529337 6819 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:30:39 dist-test-slave-khd3 krb5kdc[6806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440239, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:30:39.532857 6819 init.cc:377] Logged in from keytab as kudu/127.6.33.62@KRBTEST.COM (short username kudu)
I20260926 16:30:39.534613 6819 webserver.cc:460] Webserver started at http://127.6.33.62:38749/ using document root <none> and password file <none>
I20260926 16:30:39.535372 6819 fs_manager.cc:359] Metadata directory not provided
I20260926 16:30:39.535531 6819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:30:39.536125 6819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:30:39.539474 6819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/minicluster-data/master-0/data/instance:
uuid: "1858e3d7d1da4dfcab17289d540d5db2"
format_stamp: "Formatted at 2026-09-26 16:30:39 on dist-test-slave-khd3"
server_key: "05bbb45d950ea463344442a9b04cc58e"
server_key_iv: "7ce90f7c09ffa47edb09b01bceec6754"
server_key_version: "encryptionkey@0"
I20260926 16:30:39.540287 6819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/minicluster-data/master-0/wal/instance:
uuid: "1858e3d7d1da4dfcab17289d540d5db2"
format_stamp: "Formatted at 2026-09-26 16:30:39 on dist-test-slave-khd3"
server_key: "05bbb45d950ea463344442a9b04cc58e"
server_key_iv: "7ce90f7c09ffa47edb09b01bceec6754"
server_key_version: "encryptionkey@0"
I20260926 16:30:39.546312 6819 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260926 16:30:39.550407 6834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:30:39.552055 6819 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:30:39.552306 6819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/minicluster-data/master-0/wal
uuid: "1858e3d7d1da4dfcab17289d540d5db2"
format_stamp: "Formatted at 2026-09-26 16:30:39 on dist-test-slave-khd3"
server_key: "05bbb45d950ea463344442a9b04cc58e"
server_key_iv: "7ce90f7c09ffa47edb09b01bceec6754"
server_key_version: "encryptionkey@0"
I20260926 16:30:39.552541 6819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:30:39.582891 6819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:30:39.588078 6819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:30:39.588456 6819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:30:39.608947 6819 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.62:35023
I20260926 16:30:39.608995 6885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.62:35023 every 8 connection(s)
I20260926 16:30:39.610414 6819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
I20260926 16:30:39.614627 6886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:30:39.619339 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 6819
I20260926 16:30:39.619601 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/minicluster-data/master-0/wal/instance
I20260926 16:30:39.620419 6276 external_mini_cluster.cc:1428] Setting key 2f919e77bf248e491e6e68839a66efa4
I20260926 16:30:39.630268 6886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1858e3d7d1da4dfcab17289d540d5db2: Bootstrap starting.
I20260926 16:30:39.635555 6886 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1858e3d7d1da4dfcab17289d540d5db2: Neither blocks nor log segments found. Creating new log.
I20260926 16:30:39.636713 6886 log.cc:824] T 00000000000000000000000000000000 P 1858e3d7d1da4dfcab17289d540d5db2: Log is configured to *not* fsync() on all Append() calls
I20260926 16:30:39.639786 6886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1858e3d7d1da4dfcab17289d540d5db2: No bootstrap required, opened a new log
Sep 26 16:30:39 dist-test-slave-khd3 krb5kdc[6806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:30:39.649567 6886 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1858e3d7d1da4dfcab17289d540d5db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1858e3d7d1da4dfcab17289d540d5db2" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 35023 } }
I20260926 16:30:39.650128 6886 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1858e3d7d1da4dfcab17289d540d5db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:30:39.650247 6886 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1858e3d7d1da4dfcab17289d540d5db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1858e3d7d1da4dfcab17289d540d5db2, State: Initialized, Role: FOLLOWER
I20260926 16:30:39.651031 6886 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1858e3d7d1da4dfcab17289d540d5db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1858e3d7d1da4dfcab17289d540d5db2" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 35023 } }
I20260926 16:30:39.651376 6886 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1858e3d7d1da4dfcab17289d540d5db2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:30:39.651535 6886 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1858e3d7d1da4dfcab17289d540d5db2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:30:39.651739 6886 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1858e3d7d1da4dfcab17289d540d5db2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:30:39.654912 6886 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1858e3d7d1da4dfcab17289d540d5db2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1858e3d7d1da4dfcab17289d540d5db2" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 35023 } }
I20260926 16:30:39.655606 6886 leader_election.cc:302] T 00000000000000000000000000000000 P 1858e3d7d1da4dfcab17289d540d5db2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1858e3d7d1da4dfcab17289d540d5db2; no voters:
I20260926 16:30:39.656379 6886 leader_election.cc:288] T 00000000000000000000000000000000 P 1858e3d7d1da4dfcab17289d540d5db2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:30:39.656527 6891 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1858e3d7d1da4dfcab17289d540d5db2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:30:39.657582 6889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:39.623029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46982 (local address 127.6.33.62:35023)
0926 16:30:39.633884 (+ 10855us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:39.633907 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:39.633982 (+ 75us) server_negotiation.cc:407] Connection header received
0926 16:30:39.636221 (+ 2239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:39.636246 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:39.636688 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:39.637210 (+ 522us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:30:39.638764 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:39.640162 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:39.641614 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:39.642803 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:39.643038 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:39.647361 (+ 4323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:30:39.647373 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:30:39.647399 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:30:39.647475 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:30:39.651974 (+ 4499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:39.652941 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:39.652951 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:39.652961 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:39.653207 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:39.653654 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:39.653659 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:39.653663 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:39.654458 (+ 795us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:30:39.654796 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:39.655252 (+ 456us) server_negotiation.cc:299] Negotiation successful
0926 16:30:39.655759 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10520,"thread_start_us":10221,"threads_started":1}
I20260926 16:30:39.658393 6891 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1858e3d7d1da4dfcab17289d540d5db2 [term 1 LEADER]: Becoming Leader. State: Replica: 1858e3d7d1da4dfcab17289d540d5db2, State: Running, Role: LEADER
I20260926 16:30:39.659121 6886 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1858e3d7d1da4dfcab17289d540d5db2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:30:39.659129 6891 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1858e3d7d1da4dfcab17289d540d5db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1858e3d7d1da4dfcab17289d540d5db2" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 35023 } }
I20260926 16:30:39.663719 6893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1858e3d7d1da4dfcab17289d540d5db2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1858e3d7d1da4dfcab17289d540d5db2. Latest consensus state: current_term: 1 leader_uuid: "1858e3d7d1da4dfcab17289d540d5db2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1858e3d7d1da4dfcab17289d540d5db2" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 35023 } } }
I20260926 16:30:39.663820 6892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1858e3d7d1da4dfcab17289d540d5db2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1858e3d7d1da4dfcab17289d540d5db2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1858e3d7d1da4dfcab17289d540d5db2" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 35023 } } }
I20260926 16:30:39.664248 6893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1858e3d7d1da4dfcab17289d540d5db2 [sys.catalog]: This master's current role is: LEADER
I20260926 16:30:39.664320 6892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1858e3d7d1da4dfcab17289d540d5db2 [sys.catalog]: This master's current role is: LEADER
I20260926 16:30:39.666878 6899 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:30:39.672626 6899 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:30:39.683035 6899 catalog_manager.cc:1347] Generated new cluster ID: f957728be05c4b1eb0c63d71cf730ed3
I20260926 16:30:39.683171 6899 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:30:39.717410 6899 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:30:39.719872 6899 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:30:39.727595 6899 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1858e3d7d1da4dfcab17289d540d5db2: Generated new TSK 0
I20260926 16:30:39.728518 6899 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.33.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab.
Entry for principal kudu/127.6.33.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab.
I20260926 16:30:39.772078 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.1:0
--local_ip_for_outbound_sockets=127.6.33.1
--webserver_interface=127.6.33.1
--webserver_port=0
--tserver_master_addrs=127.6.33.62:35023
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab
--principal=kudu/127.6.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:36525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:30:40.116101 6912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:30:40.116401 6912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:30:40.116564 6912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:30:40.124210 6912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:30:40.124330 6912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:30:40.124512 6912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.1
I20260926 16:30:40.134665 6912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:36525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab
--principal=kudu/127.6.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.33.1
--webserver_port=0
--tserver_master_addrs=127.6.33.62:35023
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.1
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:30:40.136034 6912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:30:40.137506 6912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:30:40.151683 6917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:30:40.151906 6920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:30:40.151916 6918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:30:40.153110 6912 server_base.cc:1034] running on GCE node
I20260926 16:30:40.153831 6912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:30:40.154906 6912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:30:40.156162 6912 hybrid_clock.cc:648] HybridClock initialized: now 1790440240156103 us; error 48 us; skew 500 ppm
I20260926 16:30:40.156631 6912 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:30:40 dist-test-slave-khd3 krb5kdc[6806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440240, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:30:40.160513 6912 init.cc:377] Logged in from keytab as kudu/127.6.33.1@KRBTEST.COM (short username kudu)
I20260926 16:30:40.162138 6912 webserver.cc:460] Webserver started at http://127.6.33.1:34655/ using document root <none> and password file <none>
I20260926 16:30:40.162992 6912 fs_manager.cc:359] Metadata directory not provided
I20260926 16:30:40.163112 6912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:30:40.163394 6912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:30:40.166678 6912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/minicluster-data/ts-0/data/instance:
uuid: "e6adcd83895a4e2bbfb3f2a6cad0c28b"
format_stamp: "Formatted at 2026-09-26 16:30:40 on dist-test-slave-khd3"
server_key: "fc08d9a559a839548f4583bcf0b2d7fc"
server_key_iv: "4122e40fc093c4ac5560bd31498d0303"
server_key_version: "encryptionkey@0"
I20260926 16:30:40.167470 6912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/minicluster-data/ts-0/wal/instance:
uuid: "e6adcd83895a4e2bbfb3f2a6cad0c28b"
format_stamp: "Formatted at 2026-09-26 16:30:40 on dist-test-slave-khd3"
server_key: "fc08d9a559a839548f4583bcf0b2d7fc"
server_key_iv: "4122e40fc093c4ac5560bd31498d0303"
server_key_version: "encryptionkey@0"
I20260926 16:30:40.173115 6912 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 16:30:40.176541 6927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:30:40.178117 6912 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:30:40.178316 6912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/minicluster-data/ts-0/wal
uuid: "e6adcd83895a4e2bbfb3f2a6cad0c28b"
format_stamp: "Formatted at 2026-09-26 16:30:40 on dist-test-slave-khd3"
server_key: "fc08d9a559a839548f4583bcf0b2d7fc"
server_key_iv: "4122e40fc093c4ac5560bd31498d0303"
server_key_version: "encryptionkey@0"
I20260926 16:30:40.178553 6912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:30:40.200217 6912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:30:40.205204 6912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:30:40.205520 6912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:30:40.206563 6912 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:30:40.207971 6912 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:30:40.208076 6912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:30:40.208220 6912 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:30:40.208293 6912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:30:40.243563 6912 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.1:33755
I20260926 16:30:40.243660 7039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.1:33755 every 8 connection(s)
I20260926 16:30:40.244948 6912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
I20260926 16:30:40.251192 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 6912
I20260926 16:30:40.251498 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/minicluster-data/ts-0/wal/instance
I20260926 16:30:40.252377 6276 external_mini_cluster.cc:1428] Setting key d622f38f7382137ea56fa996da98fdd6
Sep 26 16:30:40 dist-test-slave-khd3 krb5kdc[6806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440240, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.1@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:30:40.269928 7042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:40.247718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:34315 (local address 127.6.33.62:35023)
0926 16:30:40.248237 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:40.248242 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:40.249282 (+ 1040us) server_negotiation.cc:407] Connection header received
0926 16:30:40.251707 (+ 2425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:40.251719 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:40.251837 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:40.251940 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:30:40.254022 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:40.255015 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:40.258021 (+ 3006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:40.258444 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:40.258555 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:40.263166 (+ 4611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:30:40.263172 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:30:40.263176 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:30:40.263211 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:30:40.266398 (+ 3187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:40.267364 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:40.267371 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:40.267374 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:40.267512 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:40.268020 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:40.268024 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:40.268026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:40.268300 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:30:40.268463 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:40.269311 (+ 848us) server_negotiation.cc:299] Negotiation successful
0926 16:30:40.269543 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":231,"threads_started":1}
WARNING: no policy specified for kudu/127.6.33.2@KRBTEST.COM; defaulting to no policy
I20260926 16:30:40.271075 7043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:40.248331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:35023 (local address 127.6.33.1:34315)
0926 16:30:40.249112 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0926 16:30:40.249159 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 16:30:40.251307 (+ 2148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:30:40.252301 (+ 994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:30:40.252314 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:30:40.253197 (+ 883us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:30:40.253844 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:40.253867 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:40.255202 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:40.255208 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:40.257860 (+ 2652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:40.257882 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:40.258703 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:40.258713 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:40.259130 (+ 417us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:40.260070 (+ 940us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:30:40.260115 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:30:40.262890 (+ 2775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:30:40.266577 (+ 3687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:30:40.266587 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:30:40.266608 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:30:40.267219 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:30:40.267632 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:30:40.267637 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:30:40.267641 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:30:40.267849 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:30:40.268443 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:30:40.268452 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:30:40.268905 (+ 453us) client_negotiation.cc:769] Sending connection context
0926 16:30:40.269186 (+ 281us) client_negotiation.cc:240] Negotiation successful
0926 16:30:40.269794 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":284,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.2@KRBTEST.COM" created.
I20260926 16:30:40.273144 7040 heartbeater.cc:347] Connected to a master server at 127.6.33.62:35023
I20260926 16:30:40.273509 7040 heartbeater.cc:464] Registering TS with master...
I20260926 16:30:40.274508 7040 heartbeater.cc:511] Master 127.6.33.62:35023 requested a full tablet report, sending...
I20260926 16:30:40.277241 6851 ts_manager.cc:194] Registered new tserver with Master: e6adcd83895a4e2bbfb3f2a6cad0c28b (127.6.33.1:33755)
I20260926 16:30:40.278853 6851 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.33.1@KRBTEST.COM'} at 127.6.33.1:34315
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab.
Entry for principal kudu/127.6.33.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab.
I20260926 16:30:40.300689 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.2:0
--local_ip_for_outbound_sockets=127.6.33.2
--webserver_interface=127.6.33.2
--webserver_port=0
--tserver_master_addrs=127.6.33.62:35023
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab
--principal=kudu/127.6.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:36525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:30:40.631968 7046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:30:40.632269 7046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:30:40.632370 7046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:30:40.639880 7046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:30:40.639999 7046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:30:40.640120 7046 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.2
I20260926 16:30:40.650413 7046 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:36525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab
--principal=kudu/127.6.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.33.2
--webserver_port=0
--tserver_master_addrs=127.6.33.62:35023
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.2
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:30:40.651705 7046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:30:40.653074 7046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:30:40.667209 7052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:30:40.667243 7054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:30:40.667378 7051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:30:40.668040 7046 server_base.cc:1034] running on GCE node
I20260926 16:30:40.668708 7046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:30:40.669762 7046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:30:40.671006 7046 hybrid_clock.cc:648] HybridClock initialized: now 1790440240670958 us; error 40 us; skew 500 ppm
I20260926 16:30:40.671376 7046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:30:40 dist-test-slave-khd3 krb5kdc[6806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440240, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:30:40.675051 7046 init.cc:377] Logged in from keytab as kudu/127.6.33.2@KRBTEST.COM (short username kudu)
I20260926 16:30:40.676573 7046 webserver.cc:460] Webserver started at http://127.6.33.2:46715/ using document root <none> and password file <none>
I20260926 16:30:40.677390 7046 fs_manager.cc:359] Metadata directory not provided
I20260926 16:30:40.677546 7046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:30:40.677865 7046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:30:40.680995 7046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/minicluster-data/ts-1/data/instance:
uuid: "903d48db20bb43f3adfc3552547204a4"
format_stamp: "Formatted at 2026-09-26 16:30:40 on dist-test-slave-khd3"
server_key: "def1f2b22861068d0a6fd4a6799edc4b"
server_key_iv: "fe01a4e1f9009eec260eab0934b4bff5"
server_key_version: "encryptionkey@0"
I20260926 16:30:40.681958 7046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/minicluster-data/ts-1/wal/instance:
uuid: "903d48db20bb43f3adfc3552547204a4"
format_stamp: "Formatted at 2026-09-26 16:30:40 on dist-test-slave-khd3"
server_key: "def1f2b22861068d0a6fd4a6799edc4b"
server_key_iv: "fe01a4e1f9009eec260eab0934b4bff5"
server_key_version: "encryptionkey@0"
I20260926 16:30:40.688009 7046 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260926 16:30:40.692148 7061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:30:40.694125 7046 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 16:30:40.694357 7046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/minicluster-data/ts-1/wal
uuid: "903d48db20bb43f3adfc3552547204a4"
format_stamp: "Formatted at 2026-09-26 16:30:40 on dist-test-slave-khd3"
server_key: "def1f2b22861068d0a6fd4a6799edc4b"
server_key_iv: "fe01a4e1f9009eec260eab0934b4bff5"
server_key_version: "encryptionkey@0"
I20260926 16:30:40.694602 7046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:30:40.711133 7046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:30:40.715739 7046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:30:40.716068 7046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:30:40.717181 7046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:30:40.718655 7046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:30:40.718787 7046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:30:40.718940 7046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:30:40.719013 7046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:30:40.753422 7046 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.2:33535
I20260926 16:30:40.753461 7173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.2:33535 every 8 connection(s)
I20260926 16:30:40.754765 7046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
I20260926 16:30:40.759404 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 7046
I20260926 16:30:40.759606 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/minicluster-data/ts-1/wal/instance
I20260926 16:30:40.760090 6276 external_mini_cluster.cc:1428] Setting key f4dbd898024b2ca72045fe8c53b4f661
Sep 26 16:30:40 dist-test-slave-khd3 krb5kdc[6806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440240, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.2@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.33.3@KRBTEST.COM; defaulting to no policy
I20260926 16:30:40.777968 7042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:40.757528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:34045 (local address 127.6.33.62:35023)
0926 16:30:40.757700 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:40.757704 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:40.759028 (+ 1324us) server_negotiation.cc:407] Connection header received
0926 16:30:40.761385 (+ 2357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:40.761391 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:40.761486 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:40.761596 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:30:40.763361 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:40.764088 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:40.765821 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:40.766569 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:40.766711 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:40.770448 (+ 3737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:30:40.770458 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:30:40.770465 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:30:40.770513 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:30:40.774500 (+ 3987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:40.775334 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:40.775340 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:40.775344 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:40.775503 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:40.775998 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:40.776004 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:40.776007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:40.776439 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:30:40.776615 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:40.777448 (+ 833us) server_negotiation.cc:299] Negotiation successful
0926 16:30:40.777707 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 16:30:40.779074 7176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:40.758076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:35023 (local address 127.6.33.2:34045)
0926 16:30:40.758783 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0926 16:30:40.758820 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 16:30:40.761098 (+ 2278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:30:40.761895 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:30:40.761912 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:30:40.762562 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:30:40.763178 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:40.763196 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:40.764261 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:40.764265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:40.765663 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:40.765676 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:40.766868 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:40.766874 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:40.767107 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:40.767849 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:30:40.767873 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:30:40.770161 (+ 2288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:30:40.774680 (+ 4519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:30:40.774688 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:30:40.774703 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:30:40.775172 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:30:40.775644 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:30:40.775647 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:30:40.775650 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:30:40.775852 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:30:40.776590 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:30:40.776596 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:30:40.777033 (+ 437us) client_negotiation.cc:769] Sending connection context
0926 16:30:40.777266 (+ 233us) client_negotiation.cc:240] Negotiation successful
0926 16:30:40.777761 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":272,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.3@KRBTEST.COM" created.
I20260926 16:30:40.780929 7174 heartbeater.cc:347] Connected to a master server at 127.6.33.62:35023
I20260926 16:30:40.781262 7174 heartbeater.cc:464] Registering TS with master...
I20260926 16:30:40.782096 7174 heartbeater.cc:511] Master 127.6.33.62:35023 requested a full tablet report, sending...
I20260926 16:30:40.783721 6851 ts_manager.cc:194] Registered new tserver with Master: 903d48db20bb43f3adfc3552547204a4 (127.6.33.2:33535)
I20260926 16:30:40.784922 6851 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.33.2@KRBTEST.COM'} at 127.6.33.2:34045
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab.
Entry for principal kudu/127.6.33.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab.
I20260926 16:30:40.805330 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.3:0
--local_ip_for_outbound_sockets=127.6.33.3
--webserver_interface=127.6.33.3
--webserver_port=0
--tserver_master_addrs=127.6.33.62:35023
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab
--principal=kudu/127.6.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:36525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:30:41.141420 7179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:30:41.141772 7179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:30:41.141947 7179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:30:41.149627 7179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:30:41.149792 7179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:30:41.150029 7179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.3
I20260926 16:30:41.160463 7179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:36525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab
--principal=kudu/127.6.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.33.3
--webserver_port=0
--tserver_master_addrs=127.6.33.62:35023
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.3
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:30:41.161895 7179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:30:41.163364 7179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:30:41.176991 7185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:30:41.177018 7184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:30:41.177073 7187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:30:41.178103 7179 server_base.cc:1034] running on GCE node
I20260926 16:30:41.178702 7179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:30:41.179723 7179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:30:41.180893 7179 hybrid_clock.cc:648] HybridClock initialized: now 1790440241180852 us; error 38 us; skew 500 ppm
I20260926 16:30:41.181182 7179 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:30:41 dist-test-slave-khd3 krb5kdc[6806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440241, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:30:41.184506 7179 init.cc:377] Logged in from keytab as kudu/127.6.33.3@KRBTEST.COM (short username kudu)
I20260926 16:30:41.186228 7179 webserver.cc:460] Webserver started at http://127.6.33.3:42767/ using document root <none> and password file <none>
I20260926 16:30:41.186936 7179 fs_manager.cc:359] Metadata directory not provided
I20260926 16:30:41.187057 7179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:30:41.187371 7179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:30:41.190397 7179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/minicluster-data/ts-2/data/instance:
uuid: "90dfeff181574b4ab47fc1c39598bc5b"
format_stamp: "Formatted at 2026-09-26 16:30:41 on dist-test-slave-khd3"
server_key: "9630e8a420717cc8b4e20d9e055fe368"
server_key_iv: "ee47ca8174cfb0747952ab08c4a3b9d0"
server_key_version: "encryptionkey@0"
I20260926 16:30:41.191237 7179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/minicluster-data/ts-2/wal/instance:
uuid: "90dfeff181574b4ab47fc1c39598bc5b"
format_stamp: "Formatted at 2026-09-26 16:30:41 on dist-test-slave-khd3"
server_key: "9630e8a420717cc8b4e20d9e055fe368"
server_key_iv: "ee47ca8174cfb0747952ab08c4a3b9d0"
server_key_version: "encryptionkey@0"
I20260926 16:30:41.196987 7179 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 16:30:41.201033 7194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:30:41.202517 7179 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:30:41.202699 7179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/minicluster-data/ts-2/wal
uuid: "90dfeff181574b4ab47fc1c39598bc5b"
format_stamp: "Formatted at 2026-09-26 16:30:41 on dist-test-slave-khd3"
server_key: "9630e8a420717cc8b4e20d9e055fe368"
server_key_iv: "ee47ca8174cfb0747952ab08c4a3b9d0"
server_key_version: "encryptionkey@0"
I20260926 16:30:41.202956 7179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:30:41.226275 7179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:30:41.231076 7179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:30:41.231393 7179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:30:41.232566 7179 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:30:41.233989 7179 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:30:41.234066 7179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:30:41.234195 7179 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:30:41.234246 7179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:30:41.268316 7179 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.3:42051
I20260926 16:30:41.268368 7306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.3:42051 every 8 connection(s)
I20260926 16:30:41.269723 7179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
I20260926 16:30:41.273372 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 7179
I20260926 16:30:41.273558 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440233850748-6276-0/minicluster-data/ts-2/wal/instance
I20260926 16:30:41.274075 6276 external_mini_cluster.cc:1428] Setting key bc1ac28e0a5b56e29ec827b42f75c942
I20260926 16:30:41.282977 7040 heartbeater.cc:503] Master 127.6.33.62:35023 was elected leader, sending a full tablet report...
Sep 26 16:30:41 dist-test-slave-khd3 krb5kdc[6806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440241, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.3@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:30:41.293556 7042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:41.272556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.3:52505 (local address 127.6.33.62:35023)
0926 16:30:41.272748 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:41.272830 (+ 82us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:41.274344 (+ 1514us) server_negotiation.cc:407] Connection header received
0926 16:30:41.276879 (+ 2535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:41.276883 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:41.276954 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:41.277043 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:30:41.278761 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:41.279544 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:41.281412 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:41.281896 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:41.282086 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:41.286221 (+ 4135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:30:41.286226 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:30:41.286230 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:30:41.286267 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:30:41.290129 (+ 3862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:41.291145 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:41.291149 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:41.291152 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:41.291307 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:41.291827 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:41.291831 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:41.291833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:41.292106 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:30:41.292218 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:41.293124 (+ 906us) server_negotiation.cc:299] Negotiation successful
0926 16:30:41.293305 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 16:30:41.294585 7309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:41.273207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:35023 (local address 127.6.33.3:52505)
0926 16:30:41.274138 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0926 16:30:41.274206 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0926 16:30:41.276464 (+ 2258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:30:41.277381 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:30:41.277395 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:30:41.277972 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:30:41.278578 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:41.278600 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:41.279750 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:41.279755 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:41.281226 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:41.281239 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:41.282193 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:41.282199 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:41.282506 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:41.283338 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:30:41.283362 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:30:41.285998 (+ 2636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:30:41.290405 (+ 4407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:30:41.290416 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:30:41.290457 (+ 41us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:30:41.291004 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:30:41.291472 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:30:41.291477 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:30:41.291479 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:30:41.291706 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:30:41.292231 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:30:41.292238 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:30:41.292628 (+ 390us) client_negotiation.cc:769] Sending connection context
0926 16:30:41.292961 (+ 333us) client_negotiation.cc:240] Negotiation successful
0926 16:30:41.293413 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":414,"threads_started":1}
I20260926 16:30:41.296028 7307 heartbeater.cc:347] Connected to a master server at 127.6.33.62:35023
I20260926 16:30:41.296401 7307 heartbeater.cc:464] Registering TS with master...
I20260926 16:30:41.297225 7307 heartbeater.cc:511] Master 127.6.33.62:35023 requested a full tablet report, sending...
I20260926 16:30:41.299000 6851 ts_manager.cc:194] Registered new tserver with Master: 90dfeff181574b4ab47fc1c39598bc5b (127.6.33.3:42051)
I20260926 16:30:41.299904 6851 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.33.3@KRBTEST.COM'} at 127.6.33.3:52505
I20260926 16:30:41.303162 6276 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:30:41.324246 7042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:41.310521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46994 (local address 127.6.33.62:35023)
0926 16:30:41.310727 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:41.310735 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:41.311204 (+ 469us) server_negotiation.cc:407] Connection header received
0926 16:30:41.311429 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:41.311439 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:41.311512 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:41.311622 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:30:41.313233 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:41.313984 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:41.315291 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:41.315718 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:41.316002 (+ 284us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:41.318686 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:30:41.318692 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:30:41.318695 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:30:41.318745 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:30:41.321477 (+ 2732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:41.322212 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:41.322218 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:41.322223 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:41.322347 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:41.322725 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:41.322732 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:41.322738 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:41.323065 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:30:41.323348 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:41.323703 (+ 355us) server_negotiation.cc:299] Negotiation successful
0926 16:30:41.323932 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 16:30:41.331028 6851 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46994:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:30:41.338451 6851 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:30:41.363054 7321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:41.351240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.3:42051 (local address 127.0.0.1:33264)
0926 16:30:41.353373 (+ 2133us) negotiation.cc:107] Waiting for socket to connect
0926 16:30:41.353401 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 16:30:41.353536 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:30:41.354594 (+ 1058us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:30:41.354602 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:30:41.354632 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:30:41.355032 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:41.355050 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:41.357130 (+ 2080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:41.357193 (+ 63us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:41.359637 (+ 2444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:41.359654 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:41.361993 (+ 2339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:41.362011 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:41.362224 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:41.362282 (+ 58us) client_negotiation.cc:769] Sending connection context
0926 16:30:41.362417 (+ 135us) client_negotiation.cc:240] Negotiation successful
0926 16:30:41.362625 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2006,"spinlock_wait_cycles":2304,"thread_start_us":1990,"threads_started":1}
I20260926 16:30:41.364133 7322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:41.351764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33264 (local address 127.6.33.3:42051)
0926 16:30:41.353364 (+ 1600us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:41.353373 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:41.353666 (+ 293us) server_negotiation.cc:407] Connection header received
0926 16:30:41.353775 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:41.353784 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:41.354058 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:41.354252 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:30:41.355246 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:41.356917 (+ 1671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:41.359866 (+ 2949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:41.361443 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:41.361572 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:41.361846 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:41.363607 (+ 1761us) server_negotiation.cc:299] Negotiation successful
0926 16:30:41.363794 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1405,"thread_start_us":1226,"threads_started":1}
I20260926 16:30:41.366235 7319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:41.351131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:33755 (local address 127.0.0.1:45760)
0926 16:30:41.351903 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0926 16:30:41.351948 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 16:30:41.352116 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:30:41.354594 (+ 2478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:30:41.354602 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:30:41.355759 (+ 1157us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:30:41.356153 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:41.356165 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:41.358153 (+ 1988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:41.358158 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:41.361852 (+ 3694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:41.361868 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:41.365372 (+ 3504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:41.365379 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:41.365554 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:41.365587 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 16:30:41.365679 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 16:30:41.365820 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":1186,"threads_started":1}
I20260926 16:30:41.366478 7320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:41.351318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45760 (local address 127.6.33.1:33755)
0926 16:30:41.353862 (+ 2544us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:41.353870 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:41.353893 (+ 23us) server_negotiation.cc:407] Connection header received
0926 16:30:41.353971 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:41.353976 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:41.354154 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:41.354338 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:30:41.356362 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:41.357993 (+ 1631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:41.362087 (+ 4094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:41.365164 (+ 3077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:41.365299 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:41.365581 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:41.365819 (+ 238us) server_negotiation.cc:299] Negotiation successful
0926 16:30:41.365993 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2385,"thread_start_us":2227,"threads_started":1}
I20260926 16:30:41.368958 7242 tablet_service.cc:1467] Processing CreateTablet for tablet c613ed8a50694c02858d63ebcac733ee (DEFAULT_TABLE table=test-table [id=951ea753c5ec47b79bcb183a4ba541a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:30:41.369943 6975 tablet_service.cc:1467] Processing CreateTablet for tablet c613ed8a50694c02858d63ebcac733ee (DEFAULT_TABLE table=test-table [id=951ea753c5ec47b79bcb183a4ba541a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:30:41.370584 7242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c613ed8a50694c02858d63ebcac733ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:30:41.371317 6975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c613ed8a50694c02858d63ebcac733ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:30:41.371809 7321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:41.353670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.2:33535 (local address 127.0.0.1:49022)
0926 16:30:41.363325 (+ 9655us) negotiation.cc:107] Waiting for socket to connect
0926 16:30:41.363344 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 16:30:41.363440 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:30:41.365819 (+ 2379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:30:41.365824 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:30:41.365853 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:30:41.366028 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:41.366038 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:41.368065 (+ 2027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:41.368070 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:41.369656 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:41.369673 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:41.371202 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:41.371208 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:41.371394 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:41.371437 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 16:30:41.371508 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 16:30:41.371603 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9544,"thread_start_us":10883,"threads_started":1}
I20260926 16:30:41.372656 7324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:41.354144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49022 (local address 127.6.33.2:33535)
0926 16:30:41.365127 (+ 10983us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:41.365136 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:41.365167 (+ 31us) server_negotiation.cc:407] Connection header received
0926 16:30:41.365272 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:41.365280 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:41.365576 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:41.366202 (+ 626us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:30:41.366560 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:41.367879 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:41.369859 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:41.371030 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:41.372142 (+ 1112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:41.372286 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:41.372344 (+ 58us) server_negotiation.cc:299] Negotiation successful
0926 16:30:41.372454 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10807,"thread_start_us":10707,"threads_started":1}
I20260926 16:30:41.375756 7109 tablet_service.cc:1467] Processing CreateTablet for tablet c613ed8a50694c02858d63ebcac733ee (DEFAULT_TABLE table=test-table [id=951ea753c5ec47b79bcb183a4ba541a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:30:41.377017 7109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c613ed8a50694c02858d63ebcac733ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:30:41.386008 7326 tablet_bootstrap.cc:492] T c613ed8a50694c02858d63ebcac733ee P 90dfeff181574b4ab47fc1c39598bc5b: Bootstrap starting.
I20260926 16:30:41.386754 7325 tablet_bootstrap.cc:492] T c613ed8a50694c02858d63ebcac733ee P e6adcd83895a4e2bbfb3f2a6cad0c28b: Bootstrap starting.
I20260926 16:30:41.388646 7327 tablet_bootstrap.cc:492] T c613ed8a50694c02858d63ebcac733ee P 903d48db20bb43f3adfc3552547204a4: Bootstrap starting.
I20260926 16:30:41.391093 7326 tablet_bootstrap.cc:654] T c613ed8a50694c02858d63ebcac733ee P 90dfeff181574b4ab47fc1c39598bc5b: Neither blocks nor log segments found. Creating new log.
I20260926 16:30:41.391232 7325 tablet_bootstrap.cc:654] T c613ed8a50694c02858d63ebcac733ee P e6adcd83895a4e2bbfb3f2a6cad0c28b: Neither blocks nor log segments found. Creating new log.
I20260926 16:30:41.392002 7327 tablet_bootstrap.cc:654] T c613ed8a50694c02858d63ebcac733ee P 903d48db20bb43f3adfc3552547204a4: Neither blocks nor log segments found. Creating new log.
I20260926 16:30:41.392359 7325 log.cc:824] T c613ed8a50694c02858d63ebcac733ee P e6adcd83895a4e2bbfb3f2a6cad0c28b: Log is configured to *not* fsync() on all Append() calls
I20260926 16:30:41.392453 7326 log.cc:824] T c613ed8a50694c02858d63ebcac733ee P 90dfeff181574b4ab47fc1c39598bc5b: Log is configured to *not* fsync() on all Append() calls
I20260926 16:30:41.393309 7327 log.cc:824] T c613ed8a50694c02858d63ebcac733ee P 903d48db20bb43f3adfc3552547204a4: Log is configured to *not* fsync() on all Append() calls
I20260926 16:30:41.395449 7326 tablet_bootstrap.cc:492] T c613ed8a50694c02858d63ebcac733ee P 90dfeff181574b4ab47fc1c39598bc5b: No bootstrap required, opened a new log
I20260926 16:30:41.395551 7325 tablet_bootstrap.cc:492] T c613ed8a50694c02858d63ebcac733ee P e6adcd83895a4e2bbfb3f2a6cad0c28b: No bootstrap required, opened a new log
I20260926 16:30:41.395737 7326 ts_tablet_manager.cc:1397] T c613ed8a50694c02858d63ebcac733ee P 90dfeff181574b4ab47fc1c39598bc5b: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260926 16:30:41.395819 7325 ts_tablet_manager.cc:1397] T c613ed8a50694c02858d63ebcac733ee P e6adcd83895a4e2bbfb3f2a6cad0c28b: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.002s
I20260926 16:30:41.396193 7327 tablet_bootstrap.cc:492] T c613ed8a50694c02858d63ebcac733ee P 903d48db20bb43f3adfc3552547204a4: No bootstrap required, opened a new log
I20260926 16:30:41.396448 7327 ts_tablet_manager.cc:1397] T c613ed8a50694c02858d63ebcac733ee P 903d48db20bb43f3adfc3552547204a4: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260926 16:30:41.404392 7326 raft_consensus.cc:348] T c613ed8a50694c02858d63ebcac733ee P 90dfeff181574b4ab47fc1c39598bc5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6adcd83895a4e2bbfb3f2a6cad0c28b" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 33755 } } peers { permanent_uuid: "90dfeff181574b4ab47fc1c39598bc5b" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 42051 } } peers { permanent_uuid: "903d48db20bb43f3adfc3552547204a4" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 33535 } }
I20260926 16:30:41.404834 7326 raft_consensus.cc:374] T c613ed8a50694c02858d63ebcac733ee P 90dfeff181574b4ab47fc1c39598bc5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:30:41.404932 7326 raft_consensus.cc:729] T c613ed8a50694c02858d63ebcac733ee P 90dfeff181574b4ab47fc1c39598bc5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90dfeff181574b4ab47fc1c39598bc5b, State: Initialized, Role: FOLLOWER
I20260926 16:30:41.405735 7326 consensus_queue.cc:260] T c613ed8a50694c02858d63ebcac733ee P 90dfeff181574b4ab47fc1c39598bc5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6adcd83895a4e2bbfb3f2a6cad0c28b" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 33755 } } peers { permanent_uuid: "90dfeff181574b4ab47fc1c39598bc5b" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 42051 } } peers { permanent_uuid: "903d48db20bb43f3adfc3552547204a4" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 33535 } }
I20260926 16:30:41.407060 7307 heartbeater.cc:503] Master 127.6.33.62:35023 was elected leader, sending a full tablet report...
I20260926 16:30:41.406872 7325 raft_consensus.cc:348] T c613ed8a50694c02858d63ebcac733ee P e6adcd83895a4e2bbfb3f2a6cad0c28b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6adcd83895a4e2bbfb3f2a6cad0c28b" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 33755 } } peers { permanent_uuid: "90dfeff181574b4ab47fc1c39598bc5b" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 42051 } } peers { permanent_uuid: "903d48db20bb43f3adfc3552547204a4" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 33535 } }
I20260926 16:30:41.407325 7325 raft_consensus.cc:374] T c613ed8a50694c02858d63ebcac733ee P e6adcd83895a4e2bbfb3f2a6cad0c28b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:30:41.407495 7325 raft_consensus.cc:729] T c613ed8a50694c02858d63ebcac733ee P e6adcd83895a4e2bbfb3f2a6cad0c28b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6adcd83895a4e2bbfb3f2a6cad0c28b, State: Initialized, Role: FOLLOWER
I20260926 16:30:41.407928 7326 ts_tablet_manager.cc:1428] T c613ed8a50694c02858d63ebcac733ee P 90dfeff181574b4ab47fc1c39598bc5b: Time spent starting tablet: real 0.012s user 0.012s sys 0.001s
I20260926 16:30:41.408107 7325 consensus_queue.cc:260] T c613ed8a50694c02858d63ebcac733ee P e6adcd83895a4e2bbfb3f2a6cad0c28b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6adcd83895a4e2bbfb3f2a6cad0c28b" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 33755 } } peers { permanent_uuid: "90dfeff181574b4ab47fc1c39598bc5b" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 42051 } } peers { permanent_uuid: "903d48db20bb43f3adfc3552547204a4" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 33535 } }
I20260926 16:30:41.408609 7327 raft_consensus.cc:348] T c613ed8a50694c02858d63ebcac733ee P 903d48db20bb43f3adfc3552547204a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6adcd83895a4e2bbfb3f2a6cad0c28b" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 33755 } } peers { permanent_uuid: "90dfeff181574b4ab47fc1c39598bc5b" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 42051 } } peers { permanent_uuid: "903d48db20bb43f3adfc3552547204a4" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 33535 } }
I20260926 16:30:41.409085 7327 raft_consensus.cc:374] T c613ed8a50694c02858d63ebcac733ee P 903d48db20bb43f3adfc3552547204a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:30:41.409190 7327 raft_consensus.cc:729] T c613ed8a50694c02858d63ebcac733ee P 903d48db20bb43f3adfc3552547204a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 903d48db20bb43f3adfc3552547204a4, State: Initialized, Role: FOLLOWER
I20260926 16:30:41.409767 7327 consensus_queue.cc:260] T c613ed8a50694c02858d63ebcac733ee P 903d48db20bb43f3adfc3552547204a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6adcd83895a4e2bbfb3f2a6cad0c28b" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 33755 } } peers { permanent_uuid: "90dfeff181574b4ab47fc1c39598bc5b" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 42051 } } peers { permanent_uuid: "903d48db20bb43f3adfc3552547204a4" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 33535 } }
I20260926 16:30:41.411038 7174 heartbeater.cc:503] Master 127.6.33.62:35023 was elected leader, sending a full tablet report...
I20260926 16:30:41.411646 7327 ts_tablet_manager.cc:1428] T c613ed8a50694c02858d63ebcac733ee P 903d48db20bb43f3adfc3552547204a4: Time spent starting tablet: real 0.015s user 0.009s sys 0.005s
I20260926 16:30:41.413116 7325 ts_tablet_manager.cc:1428] T c613ed8a50694c02858d63ebcac733ee P e6adcd83895a4e2bbfb3f2a6cad0c28b: Time spent starting tablet: real 0.017s user 0.011s sys 0.008s
W20260926 16:30:41.497718 7041 tablet.cc:2369] T c613ed8a50694c02858d63ebcac733ee P e6adcd83895a4e2bbfb3f2a6cad0c28b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:30:41.500223 7333 raft_consensus.cc:482] T c613ed8a50694c02858d63ebcac733ee P 903d48db20bb43f3adfc3552547204a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:30:41.500564 7333 raft_consensus.cc:504] T c613ed8a50694c02858d63ebcac733ee P 903d48db20bb43f3adfc3552547204a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6adcd83895a4e2bbfb3f2a6cad0c28b" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 33755 } } peers { permanent_uuid: "90dfeff181574b4ab47fc1c39598bc5b" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 42051 } } peers { permanent_uuid: "903d48db20bb43f3adfc3552547204a4" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 33535 } }
I20260926 16:30:41.502321 7333 leader_election.cc:288] T c613ed8a50694c02858d63ebcac733ee P 903d48db20bb43f3adfc3552547204a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6adcd83895a4e2bbfb3f2a6cad0c28b (127.6.33.1:33755), 90dfeff181574b4ab47fc1c39598bc5b (127.6.33.3:42051)
W20260926 16:30:41.506872 7175 tablet.cc:2369] T c613ed8a50694c02858d63ebcac733ee P 903d48db20bb43f3adfc3552547204a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:30:41.511916 7322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:41.502864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:39023 (local address 127.6.33.3:42051)
0926 16:30:41.503080 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:41.503088 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:41.503867 (+ 779us) server_negotiation.cc:407] Connection header received
0926 16:30:41.504065 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:41.504074 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:41.504133 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:41.504228 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:30:41.505117 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:41.506221 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:41.509261 (+ 3040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:41.510537 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:41.510680 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:41.510788 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:41.511524 (+ 736us) server_negotiation.cc:299] Negotiation successful
0926 16:30:41.511660 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 16:30:41.511930 7320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:41.502770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:44739 (local address 127.6.33.1:33755)
0926 16:30:41.502994 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:41.503003 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:41.503755 (+ 752us) server_negotiation.cc:407] Connection header received
0926 16:30:41.504034 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:41.504038 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:41.504097 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:41.504195 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:30:41.505005 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:41.506198 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:41.509257 (+ 3059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:41.510540 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:41.510680 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:41.510788 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:41.511524 (+ 736us) server_negotiation.cc:299] Negotiation successful
0926 16:30:41.511660 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 16:30:41.512188 7334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:41.502760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:33755 (local address 127.6.33.2:44739)
0926 16:30:41.503598 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0926 16:30:41.503615 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 16:30:41.503816 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:30:41.504209 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:30:41.504213 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:30:41.504235 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:30:41.504680 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:41.504690 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:41.506355 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:41.506361 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:41.509079 (+ 2718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:41.509095 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:41.510938 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:41.510949 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:41.511272 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:41.511369 (+ 97us) client_negotiation.cc:769] Sending connection context
0926 16:30:41.511477 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 16:30:41.511689 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"mutex_wait_us":178,"thread_start_us":466,"threads_started":1}
I20260926 16:30:41.512212 7335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:41.502761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.3:42051 (local address 127.6.33.2:39023)
0926 16:30:41.503754 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0926 16:30:41.503769 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 16:30:41.503915 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:30:41.504259 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:30:41.504263 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:30:41.504291 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:30:41.504879 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:41.504893 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:41.506362 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:41.506370 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:41.509079 (+ 2709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:41.509095 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:41.510971 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:41.510976 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:41.511248 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:41.511369 (+ 121us) client_negotiation.cc:769] Sending connection context
0926 16:30:41.511477 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 16:30:41.511689 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":877,"mutex_wait_us":295,"thread_start_us":514,"threads_started":1}
I20260926 16:30:41.513375 7262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c613ed8a50694c02858d63ebcac733ee" candidate_uuid: "903d48db20bb43f3adfc3552547204a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90dfeff181574b4ab47fc1c39598bc5b" is_pre_election: true
I20260926 16:30:41.513384 6995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c613ed8a50694c02858d63ebcac733ee" candidate_uuid: "903d48db20bb43f3adfc3552547204a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6adcd83895a4e2bbfb3f2a6cad0c28b" is_pre_election: true
I20260926 16:30:41.513995 7262 raft_consensus.cc:2454] T c613ed8a50694c02858d63ebcac733ee P 90dfeff181574b4ab47fc1c39598bc5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 903d48db20bb43f3adfc3552547204a4 in term 0.
I20260926 16:30:41.513995 6995 raft_consensus.cc:2454] T c613ed8a50694c02858d63ebcac733ee P e6adcd83895a4e2bbfb3f2a6cad0c28b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 903d48db20bb43f3adfc3552547204a4 in term 0.
I20260926 16:30:41.514810 7065 leader_election.cc:302] T c613ed8a50694c02858d63ebcac733ee P 903d48db20bb43f3adfc3552547204a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 903d48db20bb43f3adfc3552547204a4, e6adcd83895a4e2bbfb3f2a6cad0c28b; no voters:
I20260926 16:30:41.515260 7333 raft_consensus.cc:2781] T c613ed8a50694c02858d63ebcac733ee P 903d48db20bb43f3adfc3552547204a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:30:41.515409 7333 raft_consensus.cc:482] T c613ed8a50694c02858d63ebcac733ee P 903d48db20bb43f3adfc3552547204a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:30:41.515488 7333 raft_consensus.cc:3037] T c613ed8a50694c02858d63ebcac733ee P 903d48db20bb43f3adfc3552547204a4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:30:41.517663 7333 raft_consensus.cc:504] T c613ed8a50694c02858d63ebcac733ee P 903d48db20bb43f3adfc3552547204a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6adcd83895a4e2bbfb3f2a6cad0c28b" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 33755 } } peers { permanent_uuid: "90dfeff181574b4ab47fc1c39598bc5b" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 42051 } } peers { permanent_uuid: "903d48db20bb43f3adfc3552547204a4" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 33535 } }
I20260926 16:30:41.518271 7333 leader_election.cc:288] T c613ed8a50694c02858d63ebcac733ee P 903d48db20bb43f3adfc3552547204a4 [CANDIDATE]: Term 1 election: Requested vote from peers e6adcd83895a4e2bbfb3f2a6cad0c28b (127.6.33.1:33755), 90dfeff181574b4ab47fc1c39598bc5b (127.6.33.3:42051)
I20260926 16:30:41.518782 6995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c613ed8a50694c02858d63ebcac733ee" candidate_uuid: "903d48db20bb43f3adfc3552547204a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6adcd83895a4e2bbfb3f2a6cad0c28b"
I20260926 16:30:41.518846 7262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c613ed8a50694c02858d63ebcac733ee" candidate_uuid: "903d48db20bb43f3adfc3552547204a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90dfeff181574b4ab47fc1c39598bc5b"
I20260926 16:30:41.519018 6995 raft_consensus.cc:3037] T c613ed8a50694c02858d63ebcac733ee P e6adcd83895a4e2bbfb3f2a6cad0c28b [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:30:41.519095 7262 raft_consensus.cc:3037] T c613ed8a50694c02858d63ebcac733ee P 90dfeff181574b4ab47fc1c39598bc5b [term 0 FOLLOWER]: Advancing to term 1
W20260926 16:30:41.521389 7308 tablet.cc:2369] T c613ed8a50694c02858d63ebcac733ee P 90dfeff181574b4ab47fc1c39598bc5b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:30:41.521456 7262 raft_consensus.cc:2454] T c613ed8a50694c02858d63ebcac733ee P 90dfeff181574b4ab47fc1c39598bc5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 903d48db20bb43f3adfc3552547204a4 in term 1.
I20260926 16:30:41.521522 6995 raft_consensus.cc:2454] T c613ed8a50694c02858d63ebcac733ee P e6adcd83895a4e2bbfb3f2a6cad0c28b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 903d48db20bb43f3adfc3552547204a4 in term 1.
I20260926 16:30:41.521944 7064 leader_election.cc:302] T c613ed8a50694c02858d63ebcac733ee P 903d48db20bb43f3adfc3552547204a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 903d48db20bb43f3adfc3552547204a4, 90dfeff181574b4ab47fc1c39598bc5b; no voters:
I20260926 16:30:41.522298 7333 raft_consensus.cc:2781] T c613ed8a50694c02858d63ebcac733ee P 903d48db20bb43f3adfc3552547204a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:30:41.522738 7333 raft_consensus.cc:686] T c613ed8a50694c02858d63ebcac733ee P 903d48db20bb43f3adfc3552547204a4 [term 1 LEADER]: Becoming Leader. State: Replica: 903d48db20bb43f3adfc3552547204a4, State: Running, Role: LEADER
I20260926 16:30:41.523259 7333 consensus_queue.cc:237] T c613ed8a50694c02858d63ebcac733ee P 903d48db20bb43f3adfc3552547204a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6adcd83895a4e2bbfb3f2a6cad0c28b" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 33755 } } peers { permanent_uuid: "90dfeff181574b4ab47fc1c39598bc5b" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 42051 } } peers { permanent_uuid: "903d48db20bb43f3adfc3552547204a4" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 33535 } }
I20260926 16:30:41.527689 6850 catalog_manager.cc:5495] T c613ed8a50694c02858d63ebcac733ee P 903d48db20bb43f3adfc3552547204a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 903d48db20bb43f3adfc3552547204a4 (127.6.33.2). New cstate: current_term: 1 leader_uuid: "903d48db20bb43f3adfc3552547204a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6adcd83895a4e2bbfb3f2a6cad0c28b" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 33755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90dfeff181574b4ab47fc1c39598bc5b" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "903d48db20bb43f3adfc3552547204a4" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 33535 } health_report { overall_health: HEALTHY } } }
Sep 26 16:30:41 dist-test-slave-khd3 krb5kdc[6806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.1@KRBTEST.COM
I20260926 16:30:41.560218 7320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:41.544934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45770 (local address 127.6.33.1:33755)
0926 16:30:41.545111 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:41.545115 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:41.545621 (+ 506us) server_negotiation.cc:407] Connection header received
0926 16:30:41.545825 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:41.545831 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:41.545931 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:41.546052 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:30:41.547525 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:41.548241 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:41.549570 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:41.550006 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:41.550139 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:41.554216 (+ 4077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:30:41.554231 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:30:41.554246 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:30:41.554280 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:30:41.557364 (+ 3084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:41.558129 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:41.558139 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:41.558142 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:41.558314 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:41.558693 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:41.558696 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:41.558698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:41.559175 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:30:41.559411 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:41.559637 (+ 226us) server_negotiation.cc:299] Negotiation successful
0926 16:30:41.559815 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Sep 26 16:30:41 dist-test-slave-khd3 krb5kdc[6806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440241, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 16:30:41 dist-test-slave-khd3 krb5kdc[6806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440241, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.33.1@KRBTEST.COM
I20260926 16:30:41.605916 7320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:41.591874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45772 (local address 127.6.33.1:33755)
0926 16:30:41.592119 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:41.592127 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:41.592417 (+ 290us) server_negotiation.cc:407] Connection header received
0926 16:30:41.592635 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:41.592640 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:41.592705 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:41.592855 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:30:41.594189 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:41.594933 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:41.596242 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:41.596654 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:41.596809 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:41.600687 (+ 3878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:30:41.600692 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:30:41.600695 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:30:41.600727 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:30:41.603270 (+ 2543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:41.603906 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:41.603913 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:41.603916 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:41.604052 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:41.604457 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:41.604463 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:41.604466 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:41.604735 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:30:41.605037 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:41.605377 (+ 340us) server_negotiation.cc:299] Negotiation successful
0926 16:30:41.605550 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
Sep 26 16:30:41 dist-test-slave-khd3 krb5kdc[6806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440241, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 16:30:41 dist-test-slave-khd3 krb5kdc[6806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440241, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.1@KRBTEST.COM
I20260926 16:30:41.654809 7320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:41.641046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45774 (local address 127.6.33.1:33755)
0926 16:30:41.641270 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:41.641276 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:41.641617 (+ 341us) server_negotiation.cc:407] Connection header received
0926 16:30:41.641835 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:41.641840 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:41.641911 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:41.642020 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:30:41.643342 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:41.644011 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:41.645269 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:41.645702 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:41.645834 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:41.649737 (+ 3903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:30:41.649742 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:30:41.649745 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:30:41.649776 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:30:41.652165 (+ 2389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:41.652854 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:41.652859 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:41.652862 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:41.652974 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:41.653356 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:41.653360 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:41.653362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:41.653658 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:30:41.653926 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:41.654227 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 16:30:41.654477 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260926 16:30:41.656674 6955 tablet_service.cc:2908] Rejecting scan request for tablet c613ed8a50694c02858d63ebcac733ee: Uninitialized: clean time has not yet been initialized
I20260926 16:30:41.660914 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 6912
I20260926 16:30:41.674228 7320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:41.661823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45780 (local address 127.6.33.1:33755)
0926 16:30:41.662023 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:41.662028 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:41.662252 (+ 224us) server_negotiation.cc:407] Connection header received
0926 16:30:41.662466 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:41.662470 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:41.662547 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:41.662632 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:30:41.664115 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:41.664942 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:41.666206 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:41.666625 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:41.666730 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:41.669037 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:30:41.669042 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:30:41.669045 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:30:41.669076 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:30:41.671675 (+ 2599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:41.672379 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:41.672383 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:41.672385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:41.672511 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:41.673088 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:41.673092 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:41.673094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:41.673395 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:30:41.673631 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:41.673840 (+ 209us) server_negotiation.cc:299] Negotiation successful
0926 16:30:41.674006 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 16:30:41.675020 7035 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:30:41.769338 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 6912
I20260926 16:30:41.787789 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 7046
Sep 26 16:30:41 dist-test-slave-khd3 krb5kdc[6806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440241, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.2@KRBTEST.COM
I20260926 16:30:41.802531 7324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:41.788601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49028 (local address 127.6.33.2:33535)
0926 16:30:41.788869 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:41.788873 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:41.788918 (+ 45us) server_negotiation.cc:407] Connection header received
0926 16:30:41.788972 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:41.788975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:41.789042 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:41.789119 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:30:41.790358 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:41.791094 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:41.792347 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:41.792875 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:41.793002 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:41.797107 (+ 4105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:30:41.797123 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:30:41.797137 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:30:41.797180 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:30:41.800197 (+ 3017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:41.800849 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:41.800861 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:41.800867 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:41.801031 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:41.801360 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:41.801363 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:41.801365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:41.801690 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:30:41.801905 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:41.802084 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 16:30:41.802271 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 16:30:41.803207 7169 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:30:41.912317 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 7046
I20260926 16:30:41.930649 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 7179
Sep 26 16:30:41 dist-test-slave-khd3 krb5kdc[6806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440241, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.3@KRBTEST.COM
I20260926 16:30:41.946578 7322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:41.931476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33268 (local address 127.6.33.3:42051)
0926 16:30:41.931648 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:41.931652 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:41.931748 (+ 96us) server_negotiation.cc:407] Connection header received
0926 16:30:41.931962 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:41.931967 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:41.932039 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:41.932141 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:30:41.933475 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:41.934254 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:41.935584 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:41.936070 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:41.936190 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:41.940628 (+ 4438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:30:41.940645 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:30:41.940662 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:30:41.940701 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:30:41.943685 (+ 2984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:41.944461 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:41.944469 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:41.944471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:41.944612 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:41.945134 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:41.945139 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:41.945141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:41.945538 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:30:41.945773 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:41.946133 (+ 360us) server_negotiation.cc:299] Negotiation successful
0926 16:30:41.946310 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 16:30:41.947364 7302 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:30:42.038048 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 7179
I20260926 16:30:42.054873 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 6819
I20260926 16:30:42.056254 6881 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:30:42.165740 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 6819
2026-09-26T16:30:42Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4287 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:30:42 dist-test-slave-khd3 krb5kdc[7355](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:30:42 dist-test-slave-khd3 krb5kdc[7355](info): set up 2 sockets
Sep 26 16:30:42 dist-test-slave-khd3 krb5kdc[7355](info): commencing operation
krb5kdc: starting...
W20260926 16:30:43.269995 6276 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.001s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Sep 26 16:30:43 dist-test-slave-khd3 krb5kdc[7355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:30:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:30:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab.
Entry for principal kudu/127.6.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab.
I20260926 16:30:43.452504 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:39505
--webserver_interface=127.6.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:34069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.33.62:39505
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:30:43.776289 7368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:30:43.776577 7368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:30:43.776667 7368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:30:43.784080 7368 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:30:43.784194 7368 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 16:30:43.784235 7368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:30:43.784287 7368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:30:43.784337 7368 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:30:43.784389 7368 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:30:43.794373 7368 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:34069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.33.62:39505
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:39505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:30:43.795621 7368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:30:43.796954 7368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:30:43.808993 7373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:30:43.808959 7374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:30:43.808929 7376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:30:43.809823 7368 server_base.cc:1034] running on GCE node
I20260926 16:30:43.810607 7368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:30:43.812013 7368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:30:43.813271 7368 hybrid_clock.cc:648] HybridClock initialized: now 1790440243813204 us; error 44 us; skew 500 ppm
I20260926 16:30:43.813598 7368 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:30:43 dist-test-slave-khd3 krb5kdc[7355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440243, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:30:43.816668 7368 init.cc:377] Logged in from keytab as kudu/127.6.33.62@KRBTEST.COM (short username kudu)
I20260926 16:30:43.818075 7368 webserver.cc:460] Webserver started at http://127.6.33.62:45029/ using document root <none> and password file <none>
I20260926 16:30:43.818768 7368 fs_manager.cc:359] Metadata directory not provided
I20260926 16:30:43.818881 7368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:30:43.819206 7368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:30:43.822129 7368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/master-0/data/instance:
uuid: "0922caa82bf5483b82a0c8cbd01a2f88"
format_stamp: "Formatted at 2026-09-26 16:30:43 on dist-test-slave-khd3"
server_key: "5f3afbbe507c7a5c295e5c45b8ea2d53"
server_key_iv: "d53d5b1c04d693800c233831f1af5963"
server_key_version: "encryptionkey@0"
I20260926 16:30:43.822908 7368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/master-0/wal/instance:
uuid: "0922caa82bf5483b82a0c8cbd01a2f88"
format_stamp: "Formatted at 2026-09-26 16:30:43 on dist-test-slave-khd3"
server_key: "5f3afbbe507c7a5c295e5c45b8ea2d53"
server_key_iv: "d53d5b1c04d693800c233831f1af5963"
server_key_version: "encryptionkey@0"
I20260926 16:30:43.828034 7368 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 16:30:43.831385 7383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:30:43.832640 7368 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 16:30:43.832878 7368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/master-0/wal
uuid: "0922caa82bf5483b82a0c8cbd01a2f88"
format_stamp: "Formatted at 2026-09-26 16:30:43 on dist-test-slave-khd3"
server_key: "5f3afbbe507c7a5c295e5c45b8ea2d53"
server_key_iv: "d53d5b1c04d693800c233831f1af5963"
server_key_version: "encryptionkey@0"
I20260926 16:30:43.833081 7368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:30:43.854395 7368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:30:43.858889 7368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:30:43.859179 7368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:30:43.881371 7368 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.62:39505
I20260926 16:30:43.881392 7444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.62:39505 every 8 connection(s)
I20260926 16:30:43.882779 7368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
I20260926 16:30:43.886705 7445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:30:43.889776 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 7368
I20260926 16:30:43.890012 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/master-0/wal/instance
I20260926 16:30:43.890810 6276 external_mini_cluster.cc:1428] Setting key 7510d1947a5650760374766f92c00779
I20260926 16:30:43.899788 7445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0922caa82bf5483b82a0c8cbd01a2f88: Bootstrap starting.
I20260926 16:30:43.903569 7445 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0922caa82bf5483b82a0c8cbd01a2f88: Neither blocks nor log segments found. Creating new log.
I20260926 16:30:43.904930 7445 log.cc:824] T 00000000000000000000000000000000 P 0922caa82bf5483b82a0c8cbd01a2f88: Log is configured to *not* fsync() on all Append() calls
Sep 26 16:30:43 dist-test-slave-khd3 krb5kdc[7355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:30:43.907754 7445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0922caa82bf5483b82a0c8cbd01a2f88: No bootstrap required, opened a new log
I20260926 16:30:43.913995 7448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:43.893131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42622 (local address 127.6.33.62:39505)
0926 16:30:43.893926 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:43.893939 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:43.893980 (+ 41us) server_negotiation.cc:407] Connection header received
0926 16:30:43.895657 (+ 1677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:43.895679 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:43.896096 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:43.896430 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:30:43.898155 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:43.899448 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:43.901116 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:43.901782 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:43.901920 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:43.906777 (+ 4857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:30:43.906791 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:30:43.906805 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:30:43.906838 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:30:43.909984 (+ 3146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:43.910714 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:43.910721 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:43.910727 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:43.910895 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:43.911268 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:43.911277 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:43.911279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:43.911815 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:30:43.912134 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:43.912568 (+ 434us) server_negotiation.cc:299] Negotiation successful
0926 16:30:43.912990 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":382,"threads_started":1}
I20260926 16:30:43.915647 7445 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0922caa82bf5483b82a0c8cbd01a2f88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0922caa82bf5483b82a0c8cbd01a2f88" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 39505 } }
I20260926 16:30:43.916082 7445 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0922caa82bf5483b82a0c8cbd01a2f88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:30:43.916195 7445 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0922caa82bf5483b82a0c8cbd01a2f88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0922caa82bf5483b82a0c8cbd01a2f88, State: Initialized, Role: FOLLOWER
I20260926 16:30:43.916858 7445 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0922caa82bf5483b82a0c8cbd01a2f88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0922caa82bf5483b82a0c8cbd01a2f88" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 39505 } }
I20260926 16:30:43.917128 7445 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0922caa82bf5483b82a0c8cbd01a2f88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:30:43.917232 7445 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0922caa82bf5483b82a0c8cbd01a2f88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:30:43.917349 7445 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0922caa82bf5483b82a0c8cbd01a2f88 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:30:43.919519 7445 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0922caa82bf5483b82a0c8cbd01a2f88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0922caa82bf5483b82a0c8cbd01a2f88" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 39505 } }
I20260926 16:30:43.920126 7445 leader_election.cc:302] T 00000000000000000000000000000000 P 0922caa82bf5483b82a0c8cbd01a2f88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0922caa82bf5483b82a0c8cbd01a2f88; no voters:
I20260926 16:30:43.920729 7445 leader_election.cc:288] T 00000000000000000000000000000000 P 0922caa82bf5483b82a0c8cbd01a2f88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:30:43.920920 7450 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0922caa82bf5483b82a0c8cbd01a2f88 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:30:43.921748 7450 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0922caa82bf5483b82a0c8cbd01a2f88 [term 1 LEADER]: Becoming Leader. State: Replica: 0922caa82bf5483b82a0c8cbd01a2f88, State: Running, Role: LEADER
I20260926 16:30:43.922307 7450 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0922caa82bf5483b82a0c8cbd01a2f88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0922caa82bf5483b82a0c8cbd01a2f88" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 39505 } }
I20260926 16:30:43.922943 7445 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0922caa82bf5483b82a0c8cbd01a2f88 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:30:43.925158 7451 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0922caa82bf5483b82a0c8cbd01a2f88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0922caa82bf5483b82a0c8cbd01a2f88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0922caa82bf5483b82a0c8cbd01a2f88" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 39505 } } }
I20260926 16:30:43.925184 7452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0922caa82bf5483b82a0c8cbd01a2f88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0922caa82bf5483b82a0c8cbd01a2f88. Latest consensus state: current_term: 1 leader_uuid: "0922caa82bf5483b82a0c8cbd01a2f88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0922caa82bf5483b82a0c8cbd01a2f88" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 39505 } } }
I20260926 16:30:43.925680 7452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0922caa82bf5483b82a0c8cbd01a2f88 [sys.catalog]: This master's current role is: LEADER
I20260926 16:30:43.925568 7451 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0922caa82bf5483b82a0c8cbd01a2f88 [sys.catalog]: This master's current role is: LEADER
I20260926 16:30:43.927345 7455 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:30:43.932780 7455 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:30:43.941919 7455 catalog_manager.cc:1347] Generated new cluster ID: b9a483c59571438b8034f26ce7d8f416
I20260926 16:30:43.942044 7455 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:30:43.955945 7455 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:30:43.957373 7455 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:30:43.968039 7455 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0922caa82bf5483b82a0c8cbd01a2f88: Generated new TSK 0
I20260926 16:30:43.968971 7455 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.33.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab.
Entry for principal kudu/127.6.33.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab.
I20260926 16:30:44.021449 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.1:0
--local_ip_for_outbound_sockets=127.6.33.1
--webserver_interface=127.6.33.1
--webserver_port=0
--tserver_master_addrs=127.6.33.62:39505
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab
--principal=kudu/127.6.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:34069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:30:44.356371 7471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:30:44.356717 7471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:30:44.356952 7471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:30:44.364604 7471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:30:44.364809 7471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:30:44.364956 7471 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 16:30:44.365091 7471 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.1
I20260926 16:30:44.375699 7471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:34069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab
--principal=kudu/127.6.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.33.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.33.62:39505
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.1
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:30:44.377296 7471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:30:44.378664 7471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:30:44.392406 7476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:30:44.392434 7477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:30:44.392650 7479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:30:44.393375 7471 server_base.cc:1034] running on GCE node
I20260926 16:30:44.394080 7471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:30:44.395076 7471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:30:44.396291 7471 hybrid_clock.cc:648] HybridClock initialized: now 1790440244396215 us; error 57 us; skew 500 ppm
I20260926 16:30:44.396600 7471 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:30:44 dist-test-slave-khd3 krb5kdc[7355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440244, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:30:44.399926 7471 init.cc:377] Logged in from keytab as kudu/127.6.33.1@KRBTEST.COM (short username kudu)
I20260926 16:30:44.401444 7471 webserver.cc:460] Webserver started at http://127.6.33.1:40191/ using document root <none> and password file <none>
I20260926 16:30:44.402153 7471 fs_manager.cc:359] Metadata directory not provided
I20260926 16:30:44.402285 7471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:30:44.402611 7471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:30:44.405616 7471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/ts-0/data/instance:
uuid: "a7382ed5d79947958b4fef1f46e102d8"
format_stamp: "Formatted at 2026-09-26 16:30:44 on dist-test-slave-khd3"
server_key: "4c26e2fc0d9f7dfac774893a519c4e32"
server_key_iv: "aaa8ea38f93620a0fb89f30b01938878"
server_key_version: "encryptionkey@0"
I20260926 16:30:44.406479 7471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/ts-0/wal/instance:
uuid: "a7382ed5d79947958b4fef1f46e102d8"
format_stamp: "Formatted at 2026-09-26 16:30:44 on dist-test-slave-khd3"
server_key: "4c26e2fc0d9f7dfac774893a519c4e32"
server_key_iv: "aaa8ea38f93620a0fb89f30b01938878"
server_key_version: "encryptionkey@0"
I20260926 16:30:44.411680 7471 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 16:30:44.415189 7486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:30:44.416584 7471 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 16:30:44.416788 7471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/ts-0/wal
uuid: "a7382ed5d79947958b4fef1f46e102d8"
format_stamp: "Formatted at 2026-09-26 16:30:44 on dist-test-slave-khd3"
server_key: "4c26e2fc0d9f7dfac774893a519c4e32"
server_key_iv: "aaa8ea38f93620a0fb89f30b01938878"
server_key_version: "encryptionkey@0"
I20260926 16:30:44.417021 7471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:30:44.439682 7471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:30:44.444253 7471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:30:44.444494 7471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:30:44.447263 7471 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:30:44.447395 7471 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:30:44.447562 7471 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:30:44.447640 7471 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 16:30:44 dist-test-slave-khd3 krb5kdc[7355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440244, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.1@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:30:44.473531 7495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:44.448148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:36579 (local address 127.6.33.62:39505)
0926 16:30:44.448615 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:44.448622 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:44.450072 (+ 1450us) server_negotiation.cc:407] Connection header received
0926 16:30:44.453417 (+ 3345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:44.453426 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:44.453533 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:44.453688 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:30:44.456254 (+ 2566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:44.457448 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:44.459903 (+ 2455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:44.460697 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:44.460887 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:44.464889 (+ 4002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:30:44.464895 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:30:44.464900 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:30:44.464940 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:30:44.469041 (+ 4101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:44.470050 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:44.470058 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:44.470065 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:44.470277 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:44.471020 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:44.471025 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:44.471028 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:44.471462 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:30:44.471618 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:44.472928 (+ 1310us) server_negotiation.cc:299] Negotiation successful
0926 16:30:44.473159 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":244,"threads_started":1}
I20260926 16:30:44.474488 7496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:44.448910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:39505 (local address 127.6.33.1:36579)
0926 16:30:44.449819 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0926 16:30:44.449874 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 16:30:44.452852 (+ 2978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:30:44.454074 (+ 1222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:30:44.454088 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:30:44.454968 (+ 880us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:30:44.456046 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:44.456085 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:44.457611 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:44.457618 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:44.459693 (+ 2075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:44.459706 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:44.461005 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:44.461015 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:44.461301 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:44.462045 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:30:44.462073 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:30:44.464581 (+ 2508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:30:44.469201 (+ 4620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:30:44.469213 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:30:44.469241 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:30:44.469885 (+ 644us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:30:44.470428 (+ 543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:30:44.470436 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:30:44.470443 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:30:44.470810 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:30:44.471639 (+ 829us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:30:44.471653 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:30:44.472196 (+ 543us) client_negotiation.cc:769] Sending connection context
0926 16:30:44.472618 (+ 422us) client_negotiation.cc:240] Negotiation successful
0926 16:30:44.473187 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":342,"threads_started":1}
I20260926 16:30:44.496479 7495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:44.481836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:55429 (local address 127.6.33.62:39505)
0926 16:30:44.482041 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:44.482050 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:44.482757 (+ 707us) server_negotiation.cc:407] Connection header received
0926 16:30:44.482968 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:44.482973 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:44.483047 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:44.483148 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:30:44.484083 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:44.484854 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:44.486958 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:44.487594 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:44.487725 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:44.490449 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:30:44.490461 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:30:44.490466 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:30:44.490496 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:30:44.493319 (+ 2823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:44.494204 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:44.494211 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:44.494217 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:44.494369 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:44.494796 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:44.494802 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:44.494806 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:44.495231 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:30:44.495420 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:44.495916 (+ 496us) server_negotiation.cc:299] Negotiation successful
0926 16:30:44.496133 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 16:30:44.496613 7564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:44.481901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:39505 (local address 127.6.33.1:55429)
0926 16:30:44.482643 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0926 16:30:44.482663 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 16:30:44.482839 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:30:44.483172 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:30:44.483176 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:30:44.483650 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:30:44.483937 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:44.483946 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:44.485006 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:44.485010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:44.486781 (+ 1771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:44.486798 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:44.487799 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:44.487806 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:44.488051 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:44.488889 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:30:44.488921 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:30:44.490234 (+ 1313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:30:44.493460 (+ 3226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:30:44.493467 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:30:44.493470 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:30:44.494028 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:30:44.494467 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:30:44.494472 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:30:44.494476 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:30:44.494651 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:30:44.495384 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:30:44.495394 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:30:44.495639 (+ 245us) client_negotiation.cc:769] Sending connection context
0926 16:30:44.495809 (+ 170us) client_negotiation.cc:240] Negotiation successful
0926 16:30:44.496144 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":492,"threads_started":1}
I20260926 16:30:44.504853 7471 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.1:40727
I20260926 16:30:44.504890 7606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.1:40727 every 8 connection(s)
I20260926 16:30:44.506142 7471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
I20260926 16:30:44.508239 7607 heartbeater.cc:347] Connected to a master server at 127.6.33.62:39505
I20260926 16:30:44.508469 7607 heartbeater.cc:464] Registering TS with master...
I20260926 16:30:44.509166 7607 heartbeater.cc:511] Master 127.6.33.62:39505 requested a full tablet report, sending...
I20260926 16:30:44.511287 7400 ts_manager.cc:194] Registered new tserver with Master: a7382ed5d79947958b4fef1f46e102d8 (127.6.33.1:40727)
I20260926 16:30:44.511524 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 7471
I20260926 16:30:44.511775 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/ts-0/wal/instance
I20260926 16:30:44.512470 6276 external_mini_cluster.cc:1428] Setting key 660cc8d627b557d0ed5ea3107bb66418
I20260926 16:30:44.513057 7400 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.33.1@KRBTEST.COM'} at 127.6.33.1:36579
WARNING: no policy specified for kudu/127.6.33.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab.
Entry for principal kudu/127.6.33.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab.
I20260926 16:30:44.556879 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.2:0
--local_ip_for_outbound_sockets=127.6.33.2
--webserver_interface=127.6.33.2
--webserver_port=0
--tserver_master_addrs=127.6.33.62:39505
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab
--principal=kudu/127.6.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:34069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:30:44.894565 7611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:30:44.894912 7611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:30:44.895084 7611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:30:44.902619 7611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:30:44.902786 7611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:30:44.902933 7611 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 16:30:44.903075 7611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.2
I20260926 16:30:44.913050 7611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:34069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab
--principal=kudu/127.6.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.33.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.33.62:39505
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.2
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:30:44.914364 7611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:30:44.915732 7611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:30:44.929139 7619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:30:44.929323 7616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:30:44.929229 7617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:30:44.930330 7611 server_base.cc:1034] running on GCE node
I20260926 16:30:44.930971 7611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:30:44.931960 7611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:30:44.933197 7611 hybrid_clock.cc:648] HybridClock initialized: now 1790440244933135 us; error 61 us; skew 500 ppm
I20260926 16:30:44.933514 7611 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:30:44 dist-test-slave-khd3 krb5kdc[7355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440244, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:30:44.936663 7611 init.cc:377] Logged in from keytab as kudu/127.6.33.2@KRBTEST.COM (short username kudu)
I20260926 16:30:44.938311 7611 webserver.cc:460] Webserver started at http://127.6.33.2:43323/ using document root <none> and password file <none>
I20260926 16:30:44.939036 7611 fs_manager.cc:359] Metadata directory not provided
I20260926 16:30:44.939167 7611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:30:44.939493 7611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:30:44.942431 7611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/ts-1/data/instance:
uuid: "80befb9ce7ad4b58b33f3f9fcd159f02"
format_stamp: "Formatted at 2026-09-26 16:30:44 on dist-test-slave-khd3"
server_key: "4dd188d6b2bfc3d4f1a0c4246c8ba8b5"
server_key_iv: "6b6f9c193fc7cb7350e255416f890835"
server_key_version: "encryptionkey@0"
I20260926 16:30:44.943210 7611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/ts-1/wal/instance:
uuid: "80befb9ce7ad4b58b33f3f9fcd159f02"
format_stamp: "Formatted at 2026-09-26 16:30:44 on dist-test-slave-khd3"
server_key: "4dd188d6b2bfc3d4f1a0c4246c8ba8b5"
server_key_iv: "6b6f9c193fc7cb7350e255416f890835"
server_key_version: "encryptionkey@0"
I20260926 16:30:44.948381 7611 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260926 16:30:44.951845 7626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:30:44.953351 7611 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260926 16:30:44.953539 7611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/ts-1/wal
uuid: "80befb9ce7ad4b58b33f3f9fcd159f02"
format_stamp: "Formatted at 2026-09-26 16:30:44 on dist-test-slave-khd3"
server_key: "4dd188d6b2bfc3d4f1a0c4246c8ba8b5"
server_key_iv: "6b6f9c193fc7cb7350e255416f890835"
server_key_version: "encryptionkey@0"
I20260926 16:30:44.953763 7611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:30:44.975420 7611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:30:44.979959 7611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:30:44.980181 7611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:30:44.982818 7611 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:30:44.982904 7611 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:30:44.983036 7611 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:30:44.983114 7611 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 16:30:44 dist-test-slave-khd3 krb5kdc[7355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440244, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.2@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:30:45.013887 7495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:44.983965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:51283 (local address 127.6.33.62:39505)
0926 16:30:44.984173 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:44.984180 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:44.985651 (+ 1471us) server_negotiation.cc:407] Connection header received
0926 16:30:44.989420 (+ 3769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:44.989430 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:44.989545 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:44.989657 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:30:44.992354 (+ 2697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:44.993506 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:44.997286 (+ 3780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:44.998064 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:44.998240 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:45.004601 (+ 6361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:30:45.004612 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:30:45.004619 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:30:45.004674 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:30:45.008812 (+ 4138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:45.010180 (+ 1368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:45.010190 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:45.010193 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:45.010371 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:45.011093 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:45.011099 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:45.011102 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:45.011542 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:30:45.011739 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:45.013129 (+ 1390us) server_negotiation.cc:299] Negotiation successful
0926 16:30:45.013425 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 16:30:45.016526 7635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:44.984474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:39505 (local address 127.6.33.2:51283)
0926 16:30:44.985459 (+ 985us) negotiation.cc:107] Waiting for socket to connect
0926 16:30:44.985519 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 16:30:44.988983 (+ 3464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:30:44.990030 (+ 1047us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:30:44.990048 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:30:44.990949 (+ 901us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:30:44.992032 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:44.992059 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:44.993720 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:44.993727 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:44.997068 (+ 3341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:44.997088 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:44.998392 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:44.998404 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:44.998911 (+ 507us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:44.999967 (+ 1056us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:30:45.000027 (+ 60us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:30:45.004203 (+ 4176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:30:45.009057 (+ 4854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:30:45.009075 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:30:45.009105 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:30:45.009981 (+ 876us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:30:45.010555 (+ 574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:30:45.010567 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:30:45.010572 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:30:45.010924 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:30:45.011869 (+ 945us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:30:45.011885 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:30:45.012527 (+ 642us) client_negotiation.cc:769] Sending connection context
0926 16:30:45.014128 (+ 1601us) client_negotiation.cc:240] Negotiation successful
0926 16:30:45.014675 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"thread_start_us":453,"threads_started":1}
I20260926 16:30:45.023320 7611 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.2:36073
I20260926 16:30:45.023358 7744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.2:36073 every 8 connection(s)
I20260926 16:30:45.025170 7611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
I20260926 16:30:45.027642 7746 heartbeater.cc:347] Connected to a master server at 127.6.33.62:39505
I20260926 16:30:45.028069 7746 heartbeater.cc:464] Registering TS with master...
I20260926 16:30:45.029009 7746 heartbeater.cc:511] Master 127.6.33.62:39505 requested a full tablet report, sending...
I20260926 16:30:45.030745 7399 ts_manager.cc:194] Registered new tserver with Master: 80befb9ce7ad4b58b33f3f9fcd159f02 (127.6.33.2:36073)
I20260926 16:30:45.031997 7399 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.33.2@KRBTEST.COM'} at 127.6.33.2:51283
I20260926 16:30:45.034920 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 7611
I20260926 16:30:45.035126 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/ts-1/wal/instance
I20260926 16:30:45.035668 6276 external_mini_cluster.cc:1428] Setting key 67fba2fc9895e9fedb8aee0e46a1829f
I20260926 16:30:45.039963 7495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:45.024788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:49989 (local address 127.6.33.62:39505)
0926 16:30:45.024999 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:45.025006 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:45.025806 (+ 800us) server_negotiation.cc:407] Connection header received
0926 16:30:45.026046 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:45.026051 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:45.026125 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:45.026225 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:30:45.027587 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:45.028625 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:45.030158 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:45.030874 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:45.031011 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:45.033184 (+ 2173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:30:45.033190 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:30:45.033195 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:30:45.033235 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:30:45.036917 (+ 3682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:45.037806 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:45.037812 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:45.037815 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:45.037983 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:45.038396 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:45.038401 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:45.038405 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:45.038758 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:30:45.038934 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:45.039368 (+ 434us) server_negotiation.cc:299] Negotiation successful
0926 16:30:45.039595 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 16:30:45.040069 7745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:45.024869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:39505 (local address 127.6.33.2:49989)
0926 16:30:45.025689 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0926 16:30:45.025708 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 16:30:45.025886 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:30:45.026248 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:30:45.026252 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:30:45.026846 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:30:45.027462 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:45.027475 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:45.028812 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:45.028819 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:45.029971 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:45.029984 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:45.031051 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:45.031056 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:45.031171 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:45.031874 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:30:45.031901 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:30:45.032993 (+ 1092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:30:45.037052 (+ 4059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:30:45.037056 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:30:45.037059 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:30:45.037654 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:30:45.038112 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:30:45.038118 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:30:45.038122 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:30:45.038267 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:30:45.038887 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:30:45.038892 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:30:45.039084 (+ 192us) client_negotiation.cc:769] Sending connection context
0926 16:30:45.039237 (+ 153us) client_negotiation.cc:240] Negotiation successful
0926 16:30:45.039587 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":472,"threads_started":1}
WARNING: no policy specified for kudu/127.6.33.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab.
Entry for principal kudu/127.6.33.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab.
I20260926 16:30:45.080080 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.3:0
--local_ip_for_outbound_sockets=127.6.33.3
--webserver_interface=127.6.33.3
--webserver_port=0
--tserver_master_addrs=127.6.33.62:39505
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab
--principal=kudu/127.6.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:34069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:30:45.411849 7750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:30:45.412154 7750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:30:45.412250 7750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:30:45.419677 7750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:30:45.419802 7750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:30:45.419894 7750 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 16:30:45.419976 7750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.3
I20260926 16:30:45.430521 7750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:34069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab
--principal=kudu/127.6.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.33.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.33.62:39505
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.3
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:30:45.431851 7750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:30:45.433203 7750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:30:45.446831 7755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:30:45.446990 7758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:30:45.447038 7756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:30:45.448259 7750 server_base.cc:1034] running on GCE node
I20260926 16:30:45.448974 7750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:30:45.450021 7750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:30:45.451285 7750 hybrid_clock.cc:648] HybridClock initialized: now 1790440245451216 us; error 40 us; skew 500 ppm
I20260926 16:30:45.451643 7750 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:30:45 dist-test-slave-khd3 krb5kdc[7355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440245, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:30:45.455065 7750 init.cc:377] Logged in from keytab as kudu/127.6.33.3@KRBTEST.COM (short username kudu)
I20260926 16:30:45.456842 7750 webserver.cc:460] Webserver started at http://127.6.33.3:37985/ using document root <none> and password file <none>
I20260926 16:30:45.457659 7750 fs_manager.cc:359] Metadata directory not provided
I20260926 16:30:45.457813 7750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:30:45.458194 7750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:30:45.461539 7750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/ts-2/data/instance:
uuid: "5c73dc2975574aedbeb86ac18279ed45"
format_stamp: "Formatted at 2026-09-26 16:30:45 on dist-test-slave-khd3"
server_key: "755e2e25ba00178134743e9adfd25563"
server_key_iv: "2e1270bc86f6b98c73e13294ef747f71"
server_key_version: "encryptionkey@0"
I20260926 16:30:45.462404 7750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/ts-2/wal/instance:
uuid: "5c73dc2975574aedbeb86ac18279ed45"
format_stamp: "Formatted at 2026-09-26 16:30:45 on dist-test-slave-khd3"
server_key: "755e2e25ba00178134743e9adfd25563"
server_key_iv: "2e1270bc86f6b98c73e13294ef747f71"
server_key_version: "encryptionkey@0"
I20260926 16:30:45.467686 7750 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 16:30:45.471416 7765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:30:45.473078 7750 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 16:30:45.473272 7750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/ts-2/wal
uuid: "5c73dc2975574aedbeb86ac18279ed45"
format_stamp: "Formatted at 2026-09-26 16:30:45 on dist-test-slave-khd3"
server_key: "755e2e25ba00178134743e9adfd25563"
server_key_iv: "2e1270bc86f6b98c73e13294ef747f71"
server_key_version: "encryptionkey@0"
I20260926 16:30:45.473505 7750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:30:45.495932 7750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:30:45.500638 7750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:30:45.501010 7750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:30:45.503710 7750 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:30:45.503830 7750 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:30:45.503950 7750 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:30:45.504022 7750 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:30:45.516383 7607 heartbeater.cc:503] Master 127.6.33.62:39505 was elected leader, sending a full tablet report...
Sep 26 16:30:45 dist-test-slave-khd3 krb5kdc[7355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440245, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.3@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:30:45.529258 7495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:45.504587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.3:59565 (local address 127.6.33.62:39505)
0926 16:30:45.504852 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:45.504859 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:45.506321 (+ 1462us) server_negotiation.cc:407] Connection header received
0926 16:30:45.509322 (+ 3001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:45.509329 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:45.509412 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:45.509482 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:30:45.512013 (+ 2531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:45.513124 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:45.515535 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:45.516232 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:45.516381 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:45.520927 (+ 4546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:30:45.520936 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:30:45.520945 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:30:45.520990 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:30:45.525493 (+ 4503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:45.526580 (+ 1087us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:45.526586 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:45.526590 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:45.526726 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:45.527346 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:45.527353 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:45.527359 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:45.527659 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:30:45.527828 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:45.528631 (+ 803us) server_negotiation.cc:299] Negotiation successful
0926 16:30:45.528896 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 16:30:45.530369 7774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:45.505240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:39505 (local address 127.6.33.3:59565)
0926 16:30:45.506151 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0926 16:30:45.506197 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 16:30:45.508979 (+ 2782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:30:45.510036 (+ 1057us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:30:45.510052 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:30:45.510799 (+ 747us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:30:45.511800 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:45.511845 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:45.513273 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:45.513278 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:45.515326 (+ 2048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:45.515338 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:45.516519 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:45.516528 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:45.516886 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:45.518070 (+ 1184us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:30:45.518105 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:30:45.520652 (+ 2547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:30:45.525674 (+ 5022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:30:45.525688 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:30:45.525713 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:30:45.526391 (+ 678us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:30:45.526903 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:30:45.526910 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:30:45.526916 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:30:45.527220 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:30:45.527814 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:30:45.527823 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:30:45.528209 (+ 386us) client_negotiation.cc:769] Sending connection context
0926 16:30:45.528489 (+ 280us) client_negotiation.cc:240] Negotiation successful
0926 16:30:45.529097 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":365,"threads_started":1}
I20260926 16:30:45.552589 7495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:45.536287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.3:49319 (local address 127.6.33.62:39505)
0926 16:30:45.536443 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:45.536447 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:45.537308 (+ 861us) server_negotiation.cc:407] Connection header received
0926 16:30:45.537535 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:45.537539 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:45.537630 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:45.537750 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:30:45.538802 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:45.539581 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:45.541614 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:45.542401 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:45.542563 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:45.544694 (+ 2131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:30:45.544704 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:30:45.544712 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:30:45.544786 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:30:45.549146 (+ 4360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:45.550242 (+ 1096us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:45.550249 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:45.550253 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:45.550434 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:45.550935 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:45.550941 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:45.550946 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:45.551374 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:30:45.551534 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:45.552003 (+ 469us) server_negotiation.cc:299] Negotiation successful
0926 16:30:45.552235 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 16:30:45.552816 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:45.536357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:39505 (local address 127.6.33.3:49319)
0926 16:30:45.537192 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0926 16:30:45.537210 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 16:30:45.537386 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:30:45.537778 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:30:45.537782 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:30:45.538351 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:30:45.538638 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:45.538648 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:45.539740 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:45.539746 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:45.541363 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:45.541375 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:45.542710 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:45.542724 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:45.542858 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:45.543616 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:30:45.543633 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:30:45.544454 (+ 821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:30:45.549359 (+ 4905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:30:45.549367 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:30:45.549375 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:30:45.550083 (+ 708us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:30:45.550610 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:30:45.550616 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:30:45.550621 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:30:45.550796 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:30:45.551516 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:30:45.551521 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:30:45.551720 (+ 199us) client_negotiation.cc:769] Sending connection context
0926 16:30:45.551884 (+ 164us) client_negotiation.cc:240] Negotiation successful
0926 16:30:45.552277 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"thread_start_us":573,"threads_started":1}
I20260926 16:30:45.562135 7750 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.3:37879
I20260926 16:30:45.562155 7884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.3:37879 every 8 connection(s)
I20260926 16:30:45.563460 7750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
I20260926 16:30:45.565713 7885 heartbeater.cc:347] Connected to a master server at 127.6.33.62:39505
I20260926 16:30:45.566005 7885 heartbeater.cc:464] Registering TS with master...
I20260926 16:30:45.566656 7885 heartbeater.cc:511] Master 127.6.33.62:39505 requested a full tablet report, sending...
I20260926 16:30:45.568012 7399 ts_manager.cc:194] Registered new tserver with Master: 5c73dc2975574aedbeb86ac18279ed45 (127.6.33.3:37879)
I20260926 16:30:45.568955 7399 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.33.3@KRBTEST.COM'} at 127.6.33.3:59565
I20260926 16:30:45.569743 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 7750
I20260926 16:30:45.569936 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/ts-2/wal/instance
I20260926 16:30:45.570323 6276 external_mini_cluster.cc:1428] Setting key 5f74040f902a3dab1e5e14b0f5f87f49
I20260926 16:30:45.573226 6276 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 16:30:45 dist-test-slave-khd3 krb5kdc[7355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440245, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 16:30:45 dist-test-slave-khd3 krb5kdc[7355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440245, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:30:45.619828 7495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:45.605442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42630 (local address 127.6.33.62:39505)
0926 16:30:45.605692 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:45.605705 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:45.605990 (+ 285us) server_negotiation.cc:407] Connection header received
0926 16:30:45.606120 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:45.606124 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:45.606190 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:45.606320 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:30:45.607719 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:45.608499 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:45.609971 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:45.610369 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:45.610462 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:45.614341 (+ 3879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:30:45.614346 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:30:45.614350 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:30:45.614385 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:30:45.617073 (+ 2688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:45.617723 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:45.617727 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:45.617729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:45.617878 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:45.618311 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:45.618315 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:45.618317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:45.618595 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:30:45.618858 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:45.619174 (+ 316us) server_negotiation.cc:299] Negotiation successful
0926 16:30:45.619413 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260926 16:30:45.624919 7399 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42630:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:30:45.634830 7399 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:30:45.672541 7899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:45.648454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34416 (local address 127.6.33.2:36073)
0926 16:30:45.661704 (+ 13250us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:45.661718 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:45.663032 (+ 1314us) server_negotiation.cc:407] Connection header received
0926 16:30:45.663158 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:45.663170 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:45.663517 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:45.664689 (+ 1172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:30:45.664964 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:45.666792 (+ 1828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:45.669882 (+ 3090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:45.671287 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:45.671462 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:45.671687 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:45.672051 (+ 364us) server_negotiation.cc:299] Negotiation successful
0926 16:30:45.672224 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13012,"thread_start_us":12844,"threads_started":1}
I20260926 16:30:45.672578 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:45.648229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.2:36073 (local address 127.0.0.1:34416)
0926 16:30:45.662296 (+ 14067us) negotiation.cc:107] Waiting for socket to connect
0926 16:30:45.662365 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0926 16:30:45.662784 (+ 419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:30:45.663806 (+ 1022us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:30:45.663817 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:30:45.663869 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:30:45.664431 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:45.664449 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:45.666988 (+ 2539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:45.667003 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:45.669672 (+ 2669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:45.669688 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:45.671535 (+ 1847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:45.671542 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:45.671767 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:45.671844 (+ 77us) client_negotiation.cc:769] Sending connection context
0926 16:30:45.671988 (+ 144us) client_negotiation.cc:240] Negotiation successful
0926 16:30:45.672204 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13709,"thread_start_us":13519,"threads_started":1}
I20260926 16:30:45.676705 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:45.648442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.3:37879 (local address 127.0.0.1:53188)
0926 16:30:45.666380 (+ 17938us) negotiation.cc:107] Waiting for socket to connect
0926 16:30:45.666398 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 16:30:45.666500 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:30:45.668370 (+ 1870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:30:45.668376 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:30:45.668403 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:30:45.668859 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:45.668871 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:45.671049 (+ 2178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:45.671054 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:45.673991 (+ 2937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:45.674006 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:45.675933 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:45.675939 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:45.676101 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:45.676133 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 16:30:45.676221 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 16:30:45.676359 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17797,"thread_start_us":20478,"threads_started":1}
I20260926 16:30:45.677480 7902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:45.649214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53188 (local address 127.6.33.3:37879)
0926 16:30:45.667539 (+ 18325us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:45.667552 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:45.667597 (+ 45us) server_negotiation.cc:407] Connection header received
0926 16:30:45.667722 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:45.667734 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:45.668107 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:45.668370 (+ 263us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:30:45.669163 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:45.670902 (+ 1739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:45.674183 (+ 3281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:45.675445 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:45.675575 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:45.675811 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:45.676984 (+ 1173us) server_negotiation.cc:299] Negotiation successful
0926 16:30:45.677163 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18118,"thread_start_us":17898,"threads_started":1}
I20260926 16:30:45.677997 7900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:45.649011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:40727 (local address 127.0.0.1:42952)
0926 16:30:45.669206 (+ 20195us) negotiation.cc:107] Waiting for socket to connect
0926 16:30:45.669230 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 16:30:45.669348 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:30:45.670299 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:30:45.670304 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:30:45.670332 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:30:45.670636 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:45.670643 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:45.672513 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:45.672517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:45.674769 (+ 2252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:45.674782 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:45.677167 (+ 2385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:45.677174 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:45.677355 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:45.677391 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 16:30:45.677478 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 16:30:45.677763 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20053,"thread_start_us":17046,"threads_started":1}
I20260926 16:30:45.677388 7676 tablet_service.cc:1467] Processing CreateTablet for tablet 2fb4965fd5514cd0b44a6f49172508da (DEFAULT_TABLE table=test-table [id=34d51d578001481da038d9af4b07795e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:30:45.678803 7903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:45.649604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42952 (local address 127.6.33.1:40727)
0926 16:30:45.667852 (+ 18248us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:45.667862 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:45.669487 (+ 1625us) server_negotiation.cc:407] Connection header received
0926 16:30:45.669610 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:45.669619 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:45.669917 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:45.670143 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:30:45.670820 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:45.672335 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:45.674953 (+ 2618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:45.676391 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:45.676524 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:45.676720 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:45.678376 (+ 1656us) server_negotiation.cc:299] Negotiation successful
0926 16:30:45.678527 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18060,"thread_start_us":18239,"threads_started":1}
I20260926 16:30:45.679085 7676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fb4965fd5514cd0b44a6f49172508da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:30:45.681452 7815 tablet_service.cc:1467] Processing CreateTablet for tablet 2fb4965fd5514cd0b44a6f49172508da (DEFAULT_TABLE table=test-table [id=34d51d578001481da038d9af4b07795e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:30:45.682392 7537 tablet_service.cc:1467] Processing CreateTablet for tablet 2fb4965fd5514cd0b44a6f49172508da (DEFAULT_TABLE table=test-table [id=34d51d578001481da038d9af4b07795e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:30:45.683000 7815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fb4965fd5514cd0b44a6f49172508da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:30:45.683828 7537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fb4965fd5514cd0b44a6f49172508da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:30:45.691814 7904 tablet_bootstrap.cc:492] T 2fb4965fd5514cd0b44a6f49172508da P 80befb9ce7ad4b58b33f3f9fcd159f02: Bootstrap starting.
I20260926 16:30:45.695824 7904 tablet_bootstrap.cc:654] T 2fb4965fd5514cd0b44a6f49172508da P 80befb9ce7ad4b58b33f3f9fcd159f02: Neither blocks nor log segments found. Creating new log.
I20260926 16:30:45.696993 7904 log.cc:824] T 2fb4965fd5514cd0b44a6f49172508da P 80befb9ce7ad4b58b33f3f9fcd159f02: Log is configured to *not* fsync() on all Append() calls
I20260926 16:30:45.698978 7907 tablet_bootstrap.cc:492] T 2fb4965fd5514cd0b44a6f49172508da P a7382ed5d79947958b4fef1f46e102d8: Bootstrap starting.
I20260926 16:30:45.700363 7904 tablet_bootstrap.cc:492] T 2fb4965fd5514cd0b44a6f49172508da P 80befb9ce7ad4b58b33f3f9fcd159f02: No bootstrap required, opened a new log
I20260926 16:30:45.700649 7904 ts_tablet_manager.cc:1397] T 2fb4965fd5514cd0b44a6f49172508da P 80befb9ce7ad4b58b33f3f9fcd159f02: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.005s
I20260926 16:30:45.703135 7907 tablet_bootstrap.cc:654] T 2fb4965fd5514cd0b44a6f49172508da P a7382ed5d79947958b4fef1f46e102d8: Neither blocks nor log segments found. Creating new log.
I20260926 16:30:45.704497 7907 log.cc:824] T 2fb4965fd5514cd0b44a6f49172508da P a7382ed5d79947958b4fef1f46e102d8: Log is configured to *not* fsync() on all Append() calls
I20260926 16:30:45.705360 7906 tablet_bootstrap.cc:492] T 2fb4965fd5514cd0b44a6f49172508da P 5c73dc2975574aedbeb86ac18279ed45: Bootstrap starting.
I20260926 16:30:45.708276 7907 tablet_bootstrap.cc:492] T 2fb4965fd5514cd0b44a6f49172508da P a7382ed5d79947958b4fef1f46e102d8: No bootstrap required, opened a new log
I20260926 16:30:45.708560 7907 ts_tablet_manager.cc:1397] T 2fb4965fd5514cd0b44a6f49172508da P a7382ed5d79947958b4fef1f46e102d8: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260926 16:30:45.709224 7906 tablet_bootstrap.cc:654] T 2fb4965fd5514cd0b44a6f49172508da P 5c73dc2975574aedbeb86ac18279ed45: Neither blocks nor log segments found. Creating new log.
I20260926 16:30:45.710529 7906 log.cc:824] T 2fb4965fd5514cd0b44a6f49172508da P 5c73dc2975574aedbeb86ac18279ed45: Log is configured to *not* fsync() on all Append() calls
I20260926 16:30:45.711458 7904 raft_consensus.cc:348] T 2fb4965fd5514cd0b44a6f49172508da P 80befb9ce7ad4b58b33f3f9fcd159f02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80befb9ce7ad4b58b33f3f9fcd159f02" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36073 } } peers { permanent_uuid: "5c73dc2975574aedbeb86ac18279ed45" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 37879 } } peers { permanent_uuid: "a7382ed5d79947958b4fef1f46e102d8" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 40727 } }
I20260926 16:30:45.712175 7904 raft_consensus.cc:374] T 2fb4965fd5514cd0b44a6f49172508da P 80befb9ce7ad4b58b33f3f9fcd159f02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:30:45.712293 7904 raft_consensus.cc:729] T 2fb4965fd5514cd0b44a6f49172508da P 80befb9ce7ad4b58b33f3f9fcd159f02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80befb9ce7ad4b58b33f3f9fcd159f02, State: Initialized, Role: FOLLOWER
I20260926 16:30:45.713173 7904 consensus_queue.cc:260] T 2fb4965fd5514cd0b44a6f49172508da P 80befb9ce7ad4b58b33f3f9fcd159f02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80befb9ce7ad4b58b33f3f9fcd159f02" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36073 } } peers { permanent_uuid: "5c73dc2975574aedbeb86ac18279ed45" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 37879 } } peers { permanent_uuid: "a7382ed5d79947958b4fef1f46e102d8" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 40727 } }
I20260926 16:30:45.713790 7906 tablet_bootstrap.cc:492] T 2fb4965fd5514cd0b44a6f49172508da P 5c73dc2975574aedbeb86ac18279ed45: No bootstrap required, opened a new log
I20260926 16:30:45.714109 7906 ts_tablet_manager.cc:1397] T 2fb4965fd5514cd0b44a6f49172508da P 5c73dc2975574aedbeb86ac18279ed45: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.000s
I20260926 16:30:45.714483 7746 heartbeater.cc:503] Master 127.6.33.62:39505 was elected leader, sending a full tablet report...
I20260926 16:30:45.715096 7904 ts_tablet_manager.cc:1428] T 2fb4965fd5514cd0b44a6f49172508da P 80befb9ce7ad4b58b33f3f9fcd159f02: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260926 16:30:45.719370 7907 raft_consensus.cc:348] T 2fb4965fd5514cd0b44a6f49172508da P a7382ed5d79947958b4fef1f46e102d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80befb9ce7ad4b58b33f3f9fcd159f02" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36073 } } peers { permanent_uuid: "5c73dc2975574aedbeb86ac18279ed45" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 37879 } } peers { permanent_uuid: "a7382ed5d79947958b4fef1f46e102d8" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 40727 } }
I20260926 16:30:45.719838 7907 raft_consensus.cc:374] T 2fb4965fd5514cd0b44a6f49172508da P a7382ed5d79947958b4fef1f46e102d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:30:45.719945 7907 raft_consensus.cc:729] T 2fb4965fd5514cd0b44a6f49172508da P a7382ed5d79947958b4fef1f46e102d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7382ed5d79947958b4fef1f46e102d8, State: Initialized, Role: FOLLOWER
I20260926 16:30:45.720626 7907 consensus_queue.cc:260] T 2fb4965fd5514cd0b44a6f49172508da P a7382ed5d79947958b4fef1f46e102d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80befb9ce7ad4b58b33f3f9fcd159f02" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36073 } } peers { permanent_uuid: "5c73dc2975574aedbeb86ac18279ed45" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 37879 } } peers { permanent_uuid: "a7382ed5d79947958b4fef1f46e102d8" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 40727 } }
I20260926 16:30:45.723456 7907 ts_tablet_manager.cc:1428] T 2fb4965fd5514cd0b44a6f49172508da P a7382ed5d79947958b4fef1f46e102d8: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260926 16:30:45.723697 7906 raft_consensus.cc:348] T 2fb4965fd5514cd0b44a6f49172508da P 5c73dc2975574aedbeb86ac18279ed45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80befb9ce7ad4b58b33f3f9fcd159f02" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36073 } } peers { permanent_uuid: "5c73dc2975574aedbeb86ac18279ed45" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 37879 } } peers { permanent_uuid: "a7382ed5d79947958b4fef1f46e102d8" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 40727 } }
I20260926 16:30:45.724473 7906 raft_consensus.cc:374] T 2fb4965fd5514cd0b44a6f49172508da P 5c73dc2975574aedbeb86ac18279ed45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:30:45.724594 7906 raft_consensus.cc:729] T 2fb4965fd5514cd0b44a6f49172508da P 5c73dc2975574aedbeb86ac18279ed45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c73dc2975574aedbeb86ac18279ed45, State: Initialized, Role: FOLLOWER
I20260926 16:30:45.725281 7906 consensus_queue.cc:260] T 2fb4965fd5514cd0b44a6f49172508da P 5c73dc2975574aedbeb86ac18279ed45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80befb9ce7ad4b58b33f3f9fcd159f02" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36073 } } peers { permanent_uuid: "5c73dc2975574aedbeb86ac18279ed45" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 37879 } } peers { permanent_uuid: "a7382ed5d79947958b4fef1f46e102d8" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 40727 } }
I20260926 16:30:45.726446 7885 heartbeater.cc:503] Master 127.6.33.62:39505 was elected leader, sending a full tablet report...
I20260926 16:30:45.727130 7906 ts_tablet_manager.cc:1428] T 2fb4965fd5514cd0b44a6f49172508da P 5c73dc2975574aedbeb86ac18279ed45: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
W20260926 16:30:45.758909 7608 tablet.cc:2369] T 2fb4965fd5514cd0b44a6f49172508da P a7382ed5d79947958b4fef1f46e102d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:30:45.769655 7912 raft_consensus.cc:482] T 2fb4965fd5514cd0b44a6f49172508da P 5c73dc2975574aedbeb86ac18279ed45 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:30:45.770030 7912 raft_consensus.cc:504] T 2fb4965fd5514cd0b44a6f49172508da P 5c73dc2975574aedbeb86ac18279ed45 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80befb9ce7ad4b58b33f3f9fcd159f02" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36073 } } peers { permanent_uuid: "5c73dc2975574aedbeb86ac18279ed45" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 37879 } } peers { permanent_uuid: "a7382ed5d79947958b4fef1f46e102d8" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 40727 } }
I20260926 16:30:45.771776 7912 leader_election.cc:288] T 2fb4965fd5514cd0b44a6f49172508da P 5c73dc2975574aedbeb86ac18279ed45 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80befb9ce7ad4b58b33f3f9fcd159f02 (127.6.33.2:36073), a7382ed5d79947958b4fef1f46e102d8 (127.6.33.1:40727)
I20260926 16:30:45.777732 7774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:45.772215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:40727 (local address 127.6.33.3:39575)
0926 16:30:45.772565 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0926 16:30:45.772590 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 16:30:45.772897 (+ 307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:30:45.773296 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:30:45.773301 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:30:45.773328 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:30:45.773506 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:45.773514 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:45.774498 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:45.774505 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:45.775790 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:45.775801 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:45.777046 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:45.777052 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:45.777234 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:45.777261 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 16:30:45.777378 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 16:30:45.777526 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162}
I20260926 16:30:45.777825 7903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:45.772266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.3:39575 (local address 127.6.33.1:40727)
0926 16:30:45.772452 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:45.772457 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:45.772702 (+ 245us) server_negotiation.cc:407] Connection header received
0926 16:30:45.773074 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:45.773078 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:45.773149 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:45.773276 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:30:45.773640 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:45.774364 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:45.775985 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:45.776869 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:45.777043 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:45.777149 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:45.777424 (+ 275us) server_negotiation.cc:299] Negotiation successful
0926 16:30:45.777565 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260926 16:30:45.778106 7747 tablet.cc:2369] T 2fb4965fd5514cd0b44a6f49172508da P 80befb9ce7ad4b58b33f3f9fcd159f02: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:30:45.779017 7899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:45.772125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.3:34981 (local address 127.6.33.2:36073)
0926 16:30:45.772332 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:45.772341 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:45.773232 (+ 891us) server_negotiation.cc:407] Connection header received
0926 16:30:45.773442 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:45.773447 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:45.773517 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:45.773612 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:30:45.774135 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:45.774868 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:45.776627 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:45.777853 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:45.778005 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:45.778121 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:45.778691 (+ 570us) server_negotiation.cc:299] Negotiation successful
0926 16:30:45.778844 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 16:30:45.779299 7561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb4965fd5514cd0b44a6f49172508da" candidate_uuid: "5c73dc2975574aedbeb86ac18279ed45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7382ed5d79947958b4fef1f46e102d8" is_pre_election: true
I20260926 16:30:45.779551 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:45.772259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.2:36073 (local address 127.6.33.3:34981)
0926 16:30:45.773124 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0926 16:30:45.773137 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 16:30:45.773299 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:30:45.773628 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:30:45.773632 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:30:45.773654 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:30:45.773955 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:45.773964 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:45.775008 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:45.775013 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:45.776453 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:45.776464 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:45.778292 (+ 1828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:45.778301 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:45.778490 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:45.778528 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 16:30:45.779205 (+ 677us) client_negotiation.cc:240] Negotiation successful
0926 16:30:45.779303 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":742,"thread_start_us":400,"threads_started":1}
I20260926 16:30:45.779874 7561 raft_consensus.cc:2454] T 2fb4965fd5514cd0b44a6f49172508da P a7382ed5d79947958b4fef1f46e102d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c73dc2975574aedbeb86ac18279ed45 in term 0.
I20260926 16:30:45.780607 7769 leader_election.cc:302] T 2fb4965fd5514cd0b44a6f49172508da P 5c73dc2975574aedbeb86ac18279ed45 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c73dc2975574aedbeb86ac18279ed45, a7382ed5d79947958b4fef1f46e102d8; no voters:
I20260926 16:30:45.780614 7696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb4965fd5514cd0b44a6f49172508da" candidate_uuid: "5c73dc2975574aedbeb86ac18279ed45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80befb9ce7ad4b58b33f3f9fcd159f02" is_pre_election: true
I20260926 16:30:45.781080 7912 raft_consensus.cc:2781] T 2fb4965fd5514cd0b44a6f49172508da P 5c73dc2975574aedbeb86ac18279ed45 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:30:45.781152 7696 raft_consensus.cc:2454] T 2fb4965fd5514cd0b44a6f49172508da P 80befb9ce7ad4b58b33f3f9fcd159f02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c73dc2975574aedbeb86ac18279ed45 in term 0.
I20260926 16:30:45.781205 7912 raft_consensus.cc:482] T 2fb4965fd5514cd0b44a6f49172508da P 5c73dc2975574aedbeb86ac18279ed45 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:30:45.781368 7912 raft_consensus.cc:3037] T 2fb4965fd5514cd0b44a6f49172508da P 5c73dc2975574aedbeb86ac18279ed45 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:30:45.783586 7912 raft_consensus.cc:504] T 2fb4965fd5514cd0b44a6f49172508da P 5c73dc2975574aedbeb86ac18279ed45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80befb9ce7ad4b58b33f3f9fcd159f02" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36073 } } peers { permanent_uuid: "5c73dc2975574aedbeb86ac18279ed45" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 37879 } } peers { permanent_uuid: "a7382ed5d79947958b4fef1f46e102d8" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 40727 } }
I20260926 16:30:45.784206 7912 leader_election.cc:288] T 2fb4965fd5514cd0b44a6f49172508da P 5c73dc2975574aedbeb86ac18279ed45 [CANDIDATE]: Term 1 election: Requested vote from peers 80befb9ce7ad4b58b33f3f9fcd159f02 (127.6.33.2:36073), a7382ed5d79947958b4fef1f46e102d8 (127.6.33.1:40727)
I20260926 16:30:45.784639 7696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb4965fd5514cd0b44a6f49172508da" candidate_uuid: "5c73dc2975574aedbeb86ac18279ed45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80befb9ce7ad4b58b33f3f9fcd159f02"
I20260926 16:30:45.784686 7561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb4965fd5514cd0b44a6f49172508da" candidate_uuid: "5c73dc2975574aedbeb86ac18279ed45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7382ed5d79947958b4fef1f46e102d8"
I20260926 16:30:45.784919 7561 raft_consensus.cc:3037] T 2fb4965fd5514cd0b44a6f49172508da P a7382ed5d79947958b4fef1f46e102d8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:30:45.784940 7696 raft_consensus.cc:3037] T 2fb4965fd5514cd0b44a6f49172508da P 80befb9ce7ad4b58b33f3f9fcd159f02 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:30:45.787173 7561 raft_consensus.cc:2454] T 2fb4965fd5514cd0b44a6f49172508da P a7382ed5d79947958b4fef1f46e102d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c73dc2975574aedbeb86ac18279ed45 in term 1.
I20260926 16:30:45.787186 7696 raft_consensus.cc:2454] T 2fb4965fd5514cd0b44a6f49172508da P 80befb9ce7ad4b58b33f3f9fcd159f02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c73dc2975574aedbeb86ac18279ed45 in term 1.
I20260926 16:30:45.787591 7769 leader_election.cc:302] T 2fb4965fd5514cd0b44a6f49172508da P 5c73dc2975574aedbeb86ac18279ed45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c73dc2975574aedbeb86ac18279ed45, a7382ed5d79947958b4fef1f46e102d8; no voters:
I20260926 16:30:45.787925 7912 raft_consensus.cc:2781] T 2fb4965fd5514cd0b44a6f49172508da P 5c73dc2975574aedbeb86ac18279ed45 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:30:45.788375 7912 raft_consensus.cc:686] T 2fb4965fd5514cd0b44a6f49172508da P 5c73dc2975574aedbeb86ac18279ed45 [term 1 LEADER]: Becoming Leader. State: Replica: 5c73dc2975574aedbeb86ac18279ed45, State: Running, Role: LEADER
I20260926 16:30:45.789116 7912 consensus_queue.cc:237] T 2fb4965fd5514cd0b44a6f49172508da P 5c73dc2975574aedbeb86ac18279ed45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80befb9ce7ad4b58b33f3f9fcd159f02" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36073 } } peers { permanent_uuid: "5c73dc2975574aedbeb86ac18279ed45" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 37879 } } peers { permanent_uuid: "a7382ed5d79947958b4fef1f46e102d8" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 40727 } }
I20260926 16:30:45.793941 7399 catalog_manager.cc:5495] T 2fb4965fd5514cd0b44a6f49172508da P 5c73dc2975574aedbeb86ac18279ed45 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c73dc2975574aedbeb86ac18279ed45 (127.6.33.3). New cstate: current_term: 1 leader_uuid: "5c73dc2975574aedbeb86ac18279ed45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80befb9ce7ad4b58b33f3f9fcd159f02" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c73dc2975574aedbeb86ac18279ed45" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 37879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a7382ed5d79947958b4fef1f46e102d8" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 40727 } health_report { overall_health: UNKNOWN } } }
W20260926 16:30:45.815200 7886 tablet.cc:2369] T 2fb4965fd5514cd0b44a6f49172508da P 5c73dc2975574aedbeb86ac18279ed45: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:30:45.842710 7902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:45.837052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53192 (local address 127.6.33.3:37879)
0926 16:30:45.837289 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:45.837296 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:45.837396 (+ 100us) server_negotiation.cc:407] Connection header received
0926 16:30:45.837725 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:45.837731 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:45.837824 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:45.837979 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:30:45.838363 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:45.839088 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:45.840413 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:45.840890 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:45.841027 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:45.841551 (+ 524us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:30:45.842091 (+ 540us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:30:45.842259 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:45.842351 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 16:30:45.842434 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 16:30:45.850853 7561 raft_consensus.cc:1261] T 2fb4965fd5514cd0b44a6f49172508da P a7382ed5d79947958b4fef1f46e102d8 [term 1 FOLLOWER]: Refusing update from remote peer 5c73dc2975574aedbeb86ac18279ed45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:30:45.850853 7696 raft_consensus.cc:1261] T 2fb4965fd5514cd0b44a6f49172508da P 80befb9ce7ad4b58b33f3f9fcd159f02 [term 1 FOLLOWER]: Refusing update from remote peer 5c73dc2975574aedbeb86ac18279ed45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:30:45.851962 7914 consensus_queue.cc:1035] T 2fb4965fd5514cd0b44a6f49172508da P 5c73dc2975574aedbeb86ac18279ed45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80befb9ce7ad4b58b33f3f9fcd159f02" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:30:45.852327 7912 consensus_queue.cc:1035] T 2fb4965fd5514cd0b44a6f49172508da P 5c73dc2975574aedbeb86ac18279ed45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7382ed5d79947958b4fef1f46e102d8" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 40727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:30:45.888470 7398 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:30:45.888844 7398 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:30:45.893399 7398 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0922caa82bf5483b82a0c8cbd01a2f88: Sending DeleteTablet for 3 replicas of tablet 2fb4965fd5514cd0b44a6f49172508da
I20260926 16:30:45.894668 7676 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fb4965fd5514cd0b44a6f49172508da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:30:45 UTC) from {username='kudu', principal='kudu/127.6.33.62@KRBTEST.COM'} at 127.0.0.1:34416
I20260926 16:30:45.894919 7537 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fb4965fd5514cd0b44a6f49172508da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:30:45 UTC) from {username='kudu', principal='kudu/127.6.33.62@KRBTEST.COM'} at 127.0.0.1:42952
I20260926 16:30:45.895900 7815 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fb4965fd5514cd0b44a6f49172508da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:30:45 UTC) from {username='kudu', principal='kudu/127.6.33.62@KRBTEST.COM'} at 127.0.0.1:53188
I20260926 16:30:45.897327 7398 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42630:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:30:45.898854 7398 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:30:45.899186 7928 tablet_replica.cc:330] T 2fb4965fd5514cd0b44a6f49172508da P 5c73dc2975574aedbeb86ac18279ed45: stopping tablet replica
I20260926 16:30:45.899674 7928 raft_consensus.cc:2229] T 2fb4965fd5514cd0b44a6f49172508da P 5c73dc2975574aedbeb86ac18279ed45 [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:30:45.900565 7928 raft_consensus.cc:2258] T 2fb4965fd5514cd0b44a6f49172508da P 5c73dc2975574aedbeb86ac18279ed45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:30:45.901644 7927 tablet_replica.cc:330] T 2fb4965fd5514cd0b44a6f49172508da P a7382ed5d79947958b4fef1f46e102d8: stopping tablet replica
I20260926 16:30:45.902199 7927 raft_consensus.cc:2229] T 2fb4965fd5514cd0b44a6f49172508da P a7382ed5d79947958b4fef1f46e102d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:30:45.902726 7927 raft_consensus.cc:2258] T 2fb4965fd5514cd0b44a6f49172508da P a7382ed5d79947958b4fef1f46e102d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:30:45.903527 7926 tablet_replica.cc:330] T 2fb4965fd5514cd0b44a6f49172508da P 80befb9ce7ad4b58b33f3f9fcd159f02: stopping tablet replica
I20260926 16:30:45.903959 7926 raft_consensus.cc:2229] T 2fb4965fd5514cd0b44a6f49172508da P 80befb9ce7ad4b58b33f3f9fcd159f02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:30:45.904428 7928 ts_tablet_manager.cc:1905] T 2fb4965fd5514cd0b44a6f49172508da P 5c73dc2975574aedbeb86ac18279ed45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:30:45.904431 7926 raft_consensus.cc:2258] T 2fb4965fd5514cd0b44a6f49172508da P 80befb9ce7ad4b58b33f3f9fcd159f02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:30:45.905006 7927 ts_tablet_manager.cc:1905] T 2fb4965fd5514cd0b44a6f49172508da P a7382ed5d79947958b4fef1f46e102d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:30:45.906879 7926 ts_tablet_manager.cc:1905] T 2fb4965fd5514cd0b44a6f49172508da P 80befb9ce7ad4b58b33f3f9fcd159f02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:30:45.907925 7537 tablet_service.cc:1467] Processing CreateTablet for tablet f08ce60c785442b783f1af6519fb51c2 (DEFAULT_TABLE table=test-table [id=3e626a31f66645dda0b7ddf3ab4d2261]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:30:45.908432 7537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f08ce60c785442b783f1af6519fb51c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:30:45.908636 7815 tablet_service.cc:1467] Processing CreateTablet for tablet f08ce60c785442b783f1af6519fb51c2 (DEFAULT_TABLE table=test-table [id=3e626a31f66645dda0b7ddf3ab4d2261]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:30:45.909149 7815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f08ce60c785442b783f1af6519fb51c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:30:45.909971 7676 tablet_service.cc:1467] Processing CreateTablet for tablet f08ce60c785442b783f1af6519fb51c2 (DEFAULT_TABLE table=test-table [id=3e626a31f66645dda0b7ddf3ab4d2261]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:30:45.910369 7676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f08ce60c785442b783f1af6519fb51c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:30:45.913947 7926 ts_tablet_manager.cc:1918] T 2fb4965fd5514cd0b44a6f49172508da P 80befb9ce7ad4b58b33f3f9fcd159f02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:30:45.914124 7926 log.cc:1196] T 2fb4965fd5514cd0b44a6f49172508da P 80befb9ce7ad4b58b33f3f9fcd159f02: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/ts-1/wal/wals/2fb4965fd5514cd0b44a6f49172508da
I20260926 16:30:45.914685 7926 ts_tablet_manager.cc:1939] T 2fb4965fd5514cd0b44a6f49172508da P 80befb9ce7ad4b58b33f3f9fcd159f02: Deleting consensus metadata
I20260926 16:30:45.915937 7906 tablet_bootstrap.cc:492] T f08ce60c785442b783f1af6519fb51c2 P 5c73dc2975574aedbeb86ac18279ed45: Bootstrap starting.
I20260926 16:30:45.916548 7385 catalog_manager.cc:4841] TS 80befb9ce7ad4b58b33f3f9fcd159f02 (127.6.33.2:36073): tablet 2fb4965fd5514cd0b44a6f49172508da (table test-table [id=34d51d578001481da038d9af4b07795e]) successfully deleted
I20260926 16:30:45.917851 7906 tablet_bootstrap.cc:654] T f08ce60c785442b783f1af6519fb51c2 P 5c73dc2975574aedbeb86ac18279ed45: Neither blocks nor log segments found. Creating new log.
I20260926 16:30:45.920948 7927 ts_tablet_manager.cc:1918] T 2fb4965fd5514cd0b44a6f49172508da P a7382ed5d79947958b4fef1f46e102d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:30:45.921049 7904 tablet_bootstrap.cc:492] T f08ce60c785442b783f1af6519fb51c2 P 80befb9ce7ad4b58b33f3f9fcd159f02: Bootstrap starting.
I20260926 16:30:45.921125 7927 log.cc:1196] T 2fb4965fd5514cd0b44a6f49172508da P a7382ed5d79947958b4fef1f46e102d8: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/ts-0/wal/wals/2fb4965fd5514cd0b44a6f49172508da
I20260926 16:30:45.921838 7927 ts_tablet_manager.cc:1939] T 2fb4965fd5514cd0b44a6f49172508da P a7382ed5d79947958b4fef1f46e102d8: Deleting consensus metadata
I20260926 16:30:45.923049 7904 tablet_bootstrap.cc:654] T f08ce60c785442b783f1af6519fb51c2 P 80befb9ce7ad4b58b33f3f9fcd159f02: Neither blocks nor log segments found. Creating new log.
I20260926 16:30:45.923875 7387 catalog_manager.cc:4841] TS a7382ed5d79947958b4fef1f46e102d8 (127.6.33.1:40727): tablet 2fb4965fd5514cd0b44a6f49172508da (table test-table [id=34d51d578001481da038d9af4b07795e]) successfully deleted
I20260926 16:30:45.924062 7907 tablet_bootstrap.cc:492] T f08ce60c785442b783f1af6519fb51c2 P a7382ed5d79947958b4fef1f46e102d8: Bootstrap starting.
I20260926 16:30:45.924233 7906 tablet_bootstrap.cc:492] T f08ce60c785442b783f1af6519fb51c2 P 5c73dc2975574aedbeb86ac18279ed45: No bootstrap required, opened a new log
I20260926 16:30:45.924422 7906 ts_tablet_manager.cc:1397] T f08ce60c785442b783f1af6519fb51c2 P 5c73dc2975574aedbeb86ac18279ed45: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20260926 16:30:45.925659 7907 tablet_bootstrap.cc:654] T f08ce60c785442b783f1af6519fb51c2 P a7382ed5d79947958b4fef1f46e102d8: Neither blocks nor log segments found. Creating new log.
I20260926 16:30:45.925968 7906 raft_consensus.cc:348] T f08ce60c785442b783f1af6519fb51c2 P 5c73dc2975574aedbeb86ac18279ed45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7382ed5d79947958b4fef1f46e102d8" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 40727 } } peers { permanent_uuid: "80befb9ce7ad4b58b33f3f9fcd159f02" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36073 } } peers { permanent_uuid: "5c73dc2975574aedbeb86ac18279ed45" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 37879 } }
I20260926 16:30:45.926290 7906 raft_consensus.cc:374] T f08ce60c785442b783f1af6519fb51c2 P 5c73dc2975574aedbeb86ac18279ed45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:30:45.926438 7906 raft_consensus.cc:729] T f08ce60c785442b783f1af6519fb51c2 P 5c73dc2975574aedbeb86ac18279ed45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c73dc2975574aedbeb86ac18279ed45, State: Initialized, Role: FOLLOWER
I20260926 16:30:45.926882 7906 consensus_queue.cc:260] T f08ce60c785442b783f1af6519fb51c2 P 5c73dc2975574aedbeb86ac18279ed45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7382ed5d79947958b4fef1f46e102d8" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 40727 } } peers { permanent_uuid: "80befb9ce7ad4b58b33f3f9fcd159f02" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36073 } } peers { permanent_uuid: "5c73dc2975574aedbeb86ac18279ed45" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 37879 } }
I20260926 16:30:45.928368 7906 ts_tablet_manager.cc:1428] T f08ce60c785442b783f1af6519fb51c2 P 5c73dc2975574aedbeb86ac18279ed45: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 16:30:45.930122 7928 ts_tablet_manager.cc:1918] T 2fb4965fd5514cd0b44a6f49172508da P 5c73dc2975574aedbeb86ac18279ed45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:30:45.930305 7928 log.cc:1196] T 2fb4965fd5514cd0b44a6f49172508da P 5c73dc2975574aedbeb86ac18279ed45: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/ts-2/wal/wals/2fb4965fd5514cd0b44a6f49172508da
I20260926 16:30:45.931425 7928 ts_tablet_manager.cc:1939] T 2fb4965fd5514cd0b44a6f49172508da P 5c73dc2975574aedbeb86ac18279ed45: Deleting consensus metadata
I20260926 16:30:45.933180 7384 catalog_manager.cc:4841] TS 5c73dc2975574aedbeb86ac18279ed45 (127.6.33.3:37879): tablet 2fb4965fd5514cd0b44a6f49172508da (table test-table [id=34d51d578001481da038d9af4b07795e]) successfully deleted
I20260926 16:30:45.934458 7904 tablet_bootstrap.cc:492] T f08ce60c785442b783f1af6519fb51c2 P 80befb9ce7ad4b58b33f3f9fcd159f02: No bootstrap required, opened a new log
I20260926 16:30:45.934582 7907 tablet_bootstrap.cc:492] T f08ce60c785442b783f1af6519fb51c2 P a7382ed5d79947958b4fef1f46e102d8: No bootstrap required, opened a new log
I20260926 16:30:45.934635 7904 ts_tablet_manager.cc:1397] T f08ce60c785442b783f1af6519fb51c2 P 80befb9ce7ad4b58b33f3f9fcd159f02: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.003s
I20260926 16:30:45.934741 7907 ts_tablet_manager.cc:1397] T f08ce60c785442b783f1af6519fb51c2 P a7382ed5d79947958b4fef1f46e102d8: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.004s
I20260926 16:30:45.935351 7904 raft_consensus.cc:348] T f08ce60c785442b783f1af6519fb51c2 P 80befb9ce7ad4b58b33f3f9fcd159f02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7382ed5d79947958b4fef1f46e102d8" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 40727 } } peers { permanent_uuid: "80befb9ce7ad4b58b33f3f9fcd159f02" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36073 } } peers { permanent_uuid: "5c73dc2975574aedbeb86ac18279ed45" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 37879 } }
I20260926 16:30:45.935554 7904 raft_consensus.cc:374] T f08ce60c785442b783f1af6519fb51c2 P 80befb9ce7ad4b58b33f3f9fcd159f02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:30:45.935648 7904 raft_consensus.cc:729] T f08ce60c785442b783f1af6519fb51c2 P 80befb9ce7ad4b58b33f3f9fcd159f02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80befb9ce7ad4b58b33f3f9fcd159f02, State: Initialized, Role: FOLLOWER
I20260926 16:30:45.935559 7907 raft_consensus.cc:348] T f08ce60c785442b783f1af6519fb51c2 P a7382ed5d79947958b4fef1f46e102d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7382ed5d79947958b4fef1f46e102d8" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 40727 } } peers { permanent_uuid: "80befb9ce7ad4b58b33f3f9fcd159f02" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36073 } } peers { permanent_uuid: "5c73dc2975574aedbeb86ac18279ed45" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 37879 } }
I20260926 16:30:45.935770 7907 raft_consensus.cc:374] T f08ce60c785442b783f1af6519fb51c2 P a7382ed5d79947958b4fef1f46e102d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:30:45.935880 7907 raft_consensus.cc:729] T f08ce60c785442b783f1af6519fb51c2 P a7382ed5d79947958b4fef1f46e102d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7382ed5d79947958b4fef1f46e102d8, State: Initialized, Role: FOLLOWER
I20260926 16:30:45.935936 7904 consensus_queue.cc:260] T f08ce60c785442b783f1af6519fb51c2 P 80befb9ce7ad4b58b33f3f9fcd159f02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7382ed5d79947958b4fef1f46e102d8" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 40727 } } peers { permanent_uuid: "80befb9ce7ad4b58b33f3f9fcd159f02" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36073 } } peers { permanent_uuid: "5c73dc2975574aedbeb86ac18279ed45" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 37879 } }
I20260926 16:30:45.936122 7907 consensus_queue.cc:260] T f08ce60c785442b783f1af6519fb51c2 P a7382ed5d79947958b4fef1f46e102d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7382ed5d79947958b4fef1f46e102d8" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 40727 } } peers { permanent_uuid: "80befb9ce7ad4b58b33f3f9fcd159f02" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36073 } } peers { permanent_uuid: "5c73dc2975574aedbeb86ac18279ed45" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 37879 } }
I20260926 16:30:45.937088 7904 ts_tablet_manager.cc:1428] T f08ce60c785442b783f1af6519fb51c2 P 80befb9ce7ad4b58b33f3f9fcd159f02: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 16:30:45.937299 7907 ts_tablet_manager.cc:1428] T f08ce60c785442b783f1af6519fb51c2 P a7382ed5d79947958b4fef1f46e102d8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 16:30:46.067794 7912 raft_consensus.cc:482] T f08ce60c785442b783f1af6519fb51c2 P 5c73dc2975574aedbeb86ac18279ed45 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:30:46.068020 7912 raft_consensus.cc:504] T f08ce60c785442b783f1af6519fb51c2 P 5c73dc2975574aedbeb86ac18279ed45 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7382ed5d79947958b4fef1f46e102d8" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 40727 } } peers { permanent_uuid: "80befb9ce7ad4b58b33f3f9fcd159f02" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36073 } } peers { permanent_uuid: "5c73dc2975574aedbeb86ac18279ed45" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 37879 } }
I20260926 16:30:46.068838 7912 leader_election.cc:288] T f08ce60c785442b783f1af6519fb51c2 P 5c73dc2975574aedbeb86ac18279ed45 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a7382ed5d79947958b4fef1f46e102d8 (127.6.33.1:40727), 80befb9ce7ad4b58b33f3f9fcd159f02 (127.6.33.2:36073)
I20260926 16:30:46.069491 7561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f08ce60c785442b783f1af6519fb51c2" candidate_uuid: "5c73dc2975574aedbeb86ac18279ed45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7382ed5d79947958b4fef1f46e102d8" is_pre_election: true
I20260926 16:30:46.069514 7696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f08ce60c785442b783f1af6519fb51c2" candidate_uuid: "5c73dc2975574aedbeb86ac18279ed45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80befb9ce7ad4b58b33f3f9fcd159f02" is_pre_election: true
I20260926 16:30:46.069758 7561 raft_consensus.cc:2454] T f08ce60c785442b783f1af6519fb51c2 P a7382ed5d79947958b4fef1f46e102d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c73dc2975574aedbeb86ac18279ed45 in term 0.
I20260926 16:30:46.069790 7696 raft_consensus.cc:2454] T f08ce60c785442b783f1af6519fb51c2 P 80befb9ce7ad4b58b33f3f9fcd159f02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c73dc2975574aedbeb86ac18279ed45 in term 0.
I20260926 16:30:46.070281 7767 leader_election.cc:302] T f08ce60c785442b783f1af6519fb51c2 P 5c73dc2975574aedbeb86ac18279ed45 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c73dc2975574aedbeb86ac18279ed45, 80befb9ce7ad4b58b33f3f9fcd159f02; no voters:
I20260926 16:30:46.070662 7912 raft_consensus.cc:2781] T f08ce60c785442b783f1af6519fb51c2 P 5c73dc2975574aedbeb86ac18279ed45 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:30:46.070784 7912 raft_consensus.cc:482] T f08ce60c785442b783f1af6519fb51c2 P 5c73dc2975574aedbeb86ac18279ed45 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:30:46.070868 7912 raft_consensus.cc:3037] T f08ce60c785442b783f1af6519fb51c2 P 5c73dc2975574aedbeb86ac18279ed45 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:30:46.074061 7912 raft_consensus.cc:504] T f08ce60c785442b783f1af6519fb51c2 P 5c73dc2975574aedbeb86ac18279ed45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7382ed5d79947958b4fef1f46e102d8" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 40727 } } peers { permanent_uuid: "80befb9ce7ad4b58b33f3f9fcd159f02" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36073 } } peers { permanent_uuid: "5c73dc2975574aedbeb86ac18279ed45" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 37879 } }
I20260926 16:30:46.074784 7912 leader_election.cc:288] T f08ce60c785442b783f1af6519fb51c2 P 5c73dc2975574aedbeb86ac18279ed45 [CANDIDATE]: Term 1 election: Requested vote from peers a7382ed5d79947958b4fef1f46e102d8 (127.6.33.1:40727), 80befb9ce7ad4b58b33f3f9fcd159f02 (127.6.33.2:36073)
I20260926 16:30:46.075332 7561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f08ce60c785442b783f1af6519fb51c2" candidate_uuid: "5c73dc2975574aedbeb86ac18279ed45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7382ed5d79947958b4fef1f46e102d8"
I20260926 16:30:46.075398 7696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f08ce60c785442b783f1af6519fb51c2" candidate_uuid: "5c73dc2975574aedbeb86ac18279ed45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80befb9ce7ad4b58b33f3f9fcd159f02"
I20260926 16:30:46.075559 7561 raft_consensus.cc:3037] T f08ce60c785442b783f1af6519fb51c2 P a7382ed5d79947958b4fef1f46e102d8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:30:46.075608 7696 raft_consensus.cc:3037] T f08ce60c785442b783f1af6519fb51c2 P 80befb9ce7ad4b58b33f3f9fcd159f02 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:30:46.078047 7561 raft_consensus.cc:2454] T f08ce60c785442b783f1af6519fb51c2 P a7382ed5d79947958b4fef1f46e102d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c73dc2975574aedbeb86ac18279ed45 in term 1.
I20260926 16:30:46.078553 7769 leader_election.cc:302] T f08ce60c785442b783f1af6519fb51c2 P 5c73dc2975574aedbeb86ac18279ed45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c73dc2975574aedbeb86ac18279ed45, a7382ed5d79947958b4fef1f46e102d8; no voters:
I20260926 16:30:46.078614 7696 raft_consensus.cc:2454] T f08ce60c785442b783f1af6519fb51c2 P 80befb9ce7ad4b58b33f3f9fcd159f02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c73dc2975574aedbeb86ac18279ed45 in term 1.
I20260926 16:30:46.078946 7912 raft_consensus.cc:2781] T f08ce60c785442b783f1af6519fb51c2 P 5c73dc2975574aedbeb86ac18279ed45 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:30:46.079123 7912 raft_consensus.cc:686] T f08ce60c785442b783f1af6519fb51c2 P 5c73dc2975574aedbeb86ac18279ed45 [term 1 LEADER]: Becoming Leader. State: Replica: 5c73dc2975574aedbeb86ac18279ed45, State: Running, Role: LEADER
I20260926 16:30:46.079414 7912 consensus_queue.cc:237] T f08ce60c785442b783f1af6519fb51c2 P 5c73dc2975574aedbeb86ac18279ed45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7382ed5d79947958b4fef1f46e102d8" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 40727 } } peers { permanent_uuid: "80befb9ce7ad4b58b33f3f9fcd159f02" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36073 } } peers { permanent_uuid: "5c73dc2975574aedbeb86ac18279ed45" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 37879 } }
I20260926 16:30:46.082079 7398 catalog_manager.cc:5495] T f08ce60c785442b783f1af6519fb51c2 P 5c73dc2975574aedbeb86ac18279ed45 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c73dc2975574aedbeb86ac18279ed45 (127.6.33.3). New cstate: current_term: 1 leader_uuid: "5c73dc2975574aedbeb86ac18279ed45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7382ed5d79947958b4fef1f46e102d8" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 40727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80befb9ce7ad4b58b33f3f9fcd159f02" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c73dc2975574aedbeb86ac18279ed45" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 37879 } health_report { overall_health: HEALTHY } } }
Sep 26 16:30:46 dist-test-slave-khd3 krb5kdc[7355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440243, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.62@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:30:46.150226 7938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:46.135196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:39505 (local address 127.0.0.1:42646)
0926 16:30:46.135873 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0926 16:30:46.135907 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 16:30:46.136088 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:30:46.136515 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:30:46.136520 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:30:46.137392 (+ 872us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:30:46.137690 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:46.137700 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:46.138840 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:46.138848 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:46.139785 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:46.139797 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:46.140577 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:46.140583 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:46.140849 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:46.141720 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:30:46.141749 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:30:46.144032 (+ 2283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:30:46.147544 (+ 3512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:30:46.147561 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:30:46.147569 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:30:46.148015 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:30:46.148399 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:30:46.148405 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:30:46.148409 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:30:46.148556 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:30:46.149137 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:30:46.149146 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:30:46.149311 (+ 165us) client_negotiation.cc:769] Sending connection context
0926 16:30:46.149488 (+ 177us) client_negotiation.cc:240] Negotiation successful
0926 16:30:46.149799 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":335,"threads_started":1}
I20260926 16:30:46.150315 7939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:46.135281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42646 (local address 127.6.33.62:39505)
0926 16:30:46.136228 (+ 947us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:46.136233 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:46.136254 (+ 21us) server_negotiation.cc:407] Connection header received
0926 16:30:46.136307 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:46.136312 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:46.136378 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:46.136487 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:30:46.137855 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:46.138701 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:46.139927 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:46.140394 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:46.140534 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:46.144262 (+ 3728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:30:46.144267 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:30:46.144270 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:30:46.144301 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:30:46.147381 (+ 3080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:46.148144 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:46.148151 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:46.148154 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:46.148285 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:46.148661 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:46.148668 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:46.148671 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:46.149003 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:30:46.149174 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:46.149587 (+ 413us) server_negotiation.cc:299] Negotiation successful
0926 16:30:46.149954 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":145,"server-negotiator.queue_time_us":823,"thread_start_us":661,"threads_started":1}
I20260926 16:30:46.156054 7398 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.6.33.62@KRBTEST.COM'} at 127.0.0.1:42646:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 16:30:46.157558 7398 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:30:46.165086 7537 tablet_service.cc:1467] Processing CreateTablet for tablet 1222b53d8c1a419ea185a04035531b2e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=267c99b3961448de8ad3e163b1775c77]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 16:30:46.165741 7537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1222b53d8c1a419ea185a04035531b2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:30:46.169593 7676 tablet_service.cc:1467] Processing CreateTablet for tablet 1222b53d8c1a419ea185a04035531b2e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=267c99b3961448de8ad3e163b1775c77]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 16:30:46.170205 7676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1222b53d8c1a419ea185a04035531b2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:30:46.170047 7815 tablet_service.cc:1467] Processing CreateTablet for tablet 1222b53d8c1a419ea185a04035531b2e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=267c99b3961448de8ad3e163b1775c77]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 16:30:46.170586 7815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1222b53d8c1a419ea185a04035531b2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:30:46.173583 7907 tablet_bootstrap.cc:492] T 1222b53d8c1a419ea185a04035531b2e P a7382ed5d79947958b4fef1f46e102d8: Bootstrap starting.
I20260926 16:30:46.175341 7907 tablet_bootstrap.cc:654] T 1222b53d8c1a419ea185a04035531b2e P a7382ed5d79947958b4fef1f46e102d8: Neither blocks nor log segments found. Creating new log.
I20260926 16:30:46.178546 7904 tablet_bootstrap.cc:492] T 1222b53d8c1a419ea185a04035531b2e P 80befb9ce7ad4b58b33f3f9fcd159f02: Bootstrap starting.
I20260926 16:30:46.180194 7904 tablet_bootstrap.cc:654] T 1222b53d8c1a419ea185a04035531b2e P 80befb9ce7ad4b58b33f3f9fcd159f02: Neither blocks nor log segments found. Creating new log.
I20260926 16:30:46.181006 7906 tablet_bootstrap.cc:492] T 1222b53d8c1a419ea185a04035531b2e P 5c73dc2975574aedbeb86ac18279ed45: Bootstrap starting.
I20260926 16:30:46.182597 7906 tablet_bootstrap.cc:654] T 1222b53d8c1a419ea185a04035531b2e P 5c73dc2975574aedbeb86ac18279ed45: Neither blocks nor log segments found. Creating new log.
I20260926 16:30:46.186214 7907 tablet_bootstrap.cc:492] T 1222b53d8c1a419ea185a04035531b2e P a7382ed5d79947958b4fef1f46e102d8: No bootstrap required, opened a new log
I20260926 16:30:46.186501 7907 ts_tablet_manager.cc:1397] T 1222b53d8c1a419ea185a04035531b2e P a7382ed5d79947958b4fef1f46e102d8: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.010s
I20260926 16:30:46.186735 7906 tablet_bootstrap.cc:492] T 1222b53d8c1a419ea185a04035531b2e P 5c73dc2975574aedbeb86ac18279ed45: No bootstrap required, opened a new log
I20260926 16:30:46.186978 7906 ts_tablet_manager.cc:1397] T 1222b53d8c1a419ea185a04035531b2e P 5c73dc2975574aedbeb86ac18279ed45: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260926 16:30:46.187291 7907 raft_consensus.cc:348] T 1222b53d8c1a419ea185a04035531b2e P a7382ed5d79947958b4fef1f46e102d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7382ed5d79947958b4fef1f46e102d8" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 40727 } } peers { permanent_uuid: "80befb9ce7ad4b58b33f3f9fcd159f02" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36073 } } peers { permanent_uuid: "5c73dc2975574aedbeb86ac18279ed45" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 37879 } }
I20260926 16:30:46.187553 7907 raft_consensus.cc:374] T 1222b53d8c1a419ea185a04035531b2e P a7382ed5d79947958b4fef1f46e102d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:30:46.187635 7907 raft_consensus.cc:729] T 1222b53d8c1a419ea185a04035531b2e P a7382ed5d79947958b4fef1f46e102d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7382ed5d79947958b4fef1f46e102d8, State: Initialized, Role: FOLLOWER
I20260926 16:30:46.187942 7907 consensus_queue.cc:260] T 1222b53d8c1a419ea185a04035531b2e P a7382ed5d79947958b4fef1f46e102d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7382ed5d79947958b4fef1f46e102d8" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 40727 } } peers { permanent_uuid: "80befb9ce7ad4b58b33f3f9fcd159f02" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36073 } } peers { permanent_uuid: "5c73dc2975574aedbeb86ac18279ed45" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 37879 } }
I20260926 16:30:46.188378 7906 raft_consensus.cc:348] T 1222b53d8c1a419ea185a04035531b2e P 5c73dc2975574aedbeb86ac18279ed45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7382ed5d79947958b4fef1f46e102d8" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 40727 } } peers { permanent_uuid: "80befb9ce7ad4b58b33f3f9fcd159f02" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36073 } } peers { permanent_uuid: "5c73dc2975574aedbeb86ac18279ed45" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 37879 } }
I20260926 16:30:46.188423 7911 tablet_replica.cc:439] T 1222b53d8c1a419ea185a04035531b2e P a7382ed5d79947958b4fef1f46e102d8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7382ed5d79947958b4fef1f46e102d8" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 40727 } } peers { permanent_uuid: "80befb9ce7ad4b58b33f3f9fcd159f02" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36073 } } peers { permanent_uuid: "5c73dc2975574aedbeb86ac18279ed45" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 37879 } } }
I20260926 16:30:46.188740 7911 tablet_replica.cc:442] T 1222b53d8c1a419ea185a04035531b2e P a7382ed5d79947958b4fef1f46e102d8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:30:46.188916 7906 raft_consensus.cc:374] T 1222b53d8c1a419ea185a04035531b2e P 5c73dc2975574aedbeb86ac18279ed45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:30:46.189080 7906 raft_consensus.cc:729] T 1222b53d8c1a419ea185a04035531b2e P 5c73dc2975574aedbeb86ac18279ed45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c73dc2975574aedbeb86ac18279ed45, State: Initialized, Role: FOLLOWER
I20260926 16:30:46.189723 7906 consensus_queue.cc:260] T 1222b53d8c1a419ea185a04035531b2e P 5c73dc2975574aedbeb86ac18279ed45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7382ed5d79947958b4fef1f46e102d8" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 40727 } } peers { permanent_uuid: "80befb9ce7ad4b58b33f3f9fcd159f02" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36073 } } peers { permanent_uuid: "5c73dc2975574aedbeb86ac18279ed45" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 37879 } }
I20260926 16:30:46.190356 7912 tablet_replica.cc:439] T 1222b53d8c1a419ea185a04035531b2e P 5c73dc2975574aedbeb86ac18279ed45: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7382ed5d79947958b4fef1f46e102d8" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 40727 } } peers { permanent_uuid: "80befb9ce7ad4b58b33f3f9fcd159f02" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36073 } } peers { permanent_uuid: "5c73dc2975574aedbeb86ac18279ed45" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 37879 } } }
I20260926 16:30:46.190676 7912 tablet_replica.cc:442] T 1222b53d8c1a419ea185a04035531b2e P 5c73dc2975574aedbeb86ac18279ed45: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:30:46.191969 7906 ts_tablet_manager.cc:1428] T 1222b53d8c1a419ea185a04035531b2e P 5c73dc2975574aedbeb86ac18279ed45: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260926 16:30:46.193934 7907 ts_tablet_manager.cc:1428] T 1222b53d8c1a419ea185a04035531b2e P a7382ed5d79947958b4fef1f46e102d8: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20260926 16:30:46.194595 7904 tablet_bootstrap.cc:492] T 1222b53d8c1a419ea185a04035531b2e P 80befb9ce7ad4b58b33f3f9fcd159f02: No bootstrap required, opened a new log
I20260926 16:30:46.194787 7904 ts_tablet_manager.cc:1397] T 1222b53d8c1a419ea185a04035531b2e P 80befb9ce7ad4b58b33f3f9fcd159f02: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.004s
I20260926 16:30:46.195509 7904 raft_consensus.cc:348] T 1222b53d8c1a419ea185a04035531b2e P 80befb9ce7ad4b58b33f3f9fcd159f02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7382ed5d79947958b4fef1f46e102d8" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 40727 } } peers { permanent_uuid: "80befb9ce7ad4b58b33f3f9fcd159f02" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36073 } } peers { permanent_uuid: "5c73dc2975574aedbeb86ac18279ed45" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 37879 } }
I20260926 16:30:46.195922 7904 raft_consensus.cc:374] T 1222b53d8c1a419ea185a04035531b2e P 80befb9ce7ad4b58b33f3f9fcd159f02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:30:46.196033 7904 raft_consensus.cc:729] T 1222b53d8c1a419ea185a04035531b2e P 80befb9ce7ad4b58b33f3f9fcd159f02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80befb9ce7ad4b58b33f3f9fcd159f02, State: Initialized, Role: FOLLOWER
I20260926 16:30:46.196336 7904 consensus_queue.cc:260] T 1222b53d8c1a419ea185a04035531b2e P 80befb9ce7ad4b58b33f3f9fcd159f02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7382ed5d79947958b4fef1f46e102d8" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 40727 } } peers { permanent_uuid: "80befb9ce7ad4b58b33f3f9fcd159f02" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36073 } } peers { permanent_uuid: "5c73dc2975574aedbeb86ac18279ed45" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 37879 } }
I20260926 16:30:46.196820 7910 tablet_replica.cc:439] T 1222b53d8c1a419ea185a04035531b2e P 80befb9ce7ad4b58b33f3f9fcd159f02: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7382ed5d79947958b4fef1f46e102d8" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 40727 } } peers { permanent_uuid: "80befb9ce7ad4b58b33f3f9fcd159f02" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36073 } } peers { permanent_uuid: "5c73dc2975574aedbeb86ac18279ed45" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 37879 } } }
I20260926 16:30:46.197122 7910 tablet_replica.cc:442] T 1222b53d8c1a419ea185a04035531b2e P 80befb9ce7ad4b58b33f3f9fcd159f02: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:30:46.198354 7904 ts_tablet_manager.cc:1428] T 1222b53d8c1a419ea185a04035531b2e P 80befb9ce7ad4b58b33f3f9fcd159f02: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260926 16:30:46.276192 7912 raft_consensus.cc:482] T 1222b53d8c1a419ea185a04035531b2e P 5c73dc2975574aedbeb86ac18279ed45 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:30:46.276464 7912 raft_consensus.cc:504] T 1222b53d8c1a419ea185a04035531b2e P 5c73dc2975574aedbeb86ac18279ed45 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7382ed5d79947958b4fef1f46e102d8" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 40727 } } peers { permanent_uuid: "80befb9ce7ad4b58b33f3f9fcd159f02" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36073 } } peers { permanent_uuid: "5c73dc2975574aedbeb86ac18279ed45" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 37879 } }
I20260926 16:30:46.277293 7912 leader_election.cc:288] T 1222b53d8c1a419ea185a04035531b2e P 5c73dc2975574aedbeb86ac18279ed45 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a7382ed5d79947958b4fef1f46e102d8 (127.6.33.1:40727), 80befb9ce7ad4b58b33f3f9fcd159f02 (127.6.33.2:36073)
I20260926 16:30:46.277869 7561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1222b53d8c1a419ea185a04035531b2e" candidate_uuid: "5c73dc2975574aedbeb86ac18279ed45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7382ed5d79947958b4fef1f46e102d8" is_pre_election: true
I20260926 16:30:46.277956 7696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1222b53d8c1a419ea185a04035531b2e" candidate_uuid: "5c73dc2975574aedbeb86ac18279ed45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80befb9ce7ad4b58b33f3f9fcd159f02" is_pre_election: true
I20260926 16:30:46.278162 7561 raft_consensus.cc:2454] T 1222b53d8c1a419ea185a04035531b2e P a7382ed5d79947958b4fef1f46e102d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c73dc2975574aedbeb86ac18279ed45 in term 0.
I20260926 16:30:46.278252 7696 raft_consensus.cc:2454] T 1222b53d8c1a419ea185a04035531b2e P 80befb9ce7ad4b58b33f3f9fcd159f02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c73dc2975574aedbeb86ac18279ed45 in term 0.
I20260926 16:30:46.278724 7769 leader_election.cc:302] T 1222b53d8c1a419ea185a04035531b2e P 5c73dc2975574aedbeb86ac18279ed45 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c73dc2975574aedbeb86ac18279ed45, a7382ed5d79947958b4fef1f46e102d8; no voters:
I20260926 16:30:46.279036 7912 raft_consensus.cc:2781] T 1222b53d8c1a419ea185a04035531b2e P 5c73dc2975574aedbeb86ac18279ed45 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:30:46.279147 7912 raft_consensus.cc:482] T 1222b53d8c1a419ea185a04035531b2e P 5c73dc2975574aedbeb86ac18279ed45 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:30:46.279275 7912 raft_consensus.cc:3037] T 1222b53d8c1a419ea185a04035531b2e P 5c73dc2975574aedbeb86ac18279ed45 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:30:46.282622 7912 raft_consensus.cc:504] T 1222b53d8c1a419ea185a04035531b2e P 5c73dc2975574aedbeb86ac18279ed45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7382ed5d79947958b4fef1f46e102d8" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 40727 } } peers { permanent_uuid: "80befb9ce7ad4b58b33f3f9fcd159f02" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36073 } } peers { permanent_uuid: "5c73dc2975574aedbeb86ac18279ed45" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 37879 } }
I20260926 16:30:46.283318 7912 leader_election.cc:288] T 1222b53d8c1a419ea185a04035531b2e P 5c73dc2975574aedbeb86ac18279ed45 [CANDIDATE]: Term 1 election: Requested vote from peers a7382ed5d79947958b4fef1f46e102d8 (127.6.33.1:40727), 80befb9ce7ad4b58b33f3f9fcd159f02 (127.6.33.2:36073)
I20260926 16:30:46.283823 7696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1222b53d8c1a419ea185a04035531b2e" candidate_uuid: "5c73dc2975574aedbeb86ac18279ed45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80befb9ce7ad4b58b33f3f9fcd159f02"
I20260926 16:30:46.283838 7561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1222b53d8c1a419ea185a04035531b2e" candidate_uuid: "5c73dc2975574aedbeb86ac18279ed45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7382ed5d79947958b4fef1f46e102d8"
I20260926 16:30:46.284060 7696 raft_consensus.cc:3037] T 1222b53d8c1a419ea185a04035531b2e P 80befb9ce7ad4b58b33f3f9fcd159f02 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:30:46.284075 7561 raft_consensus.cc:3037] T 1222b53d8c1a419ea185a04035531b2e P a7382ed5d79947958b4fef1f46e102d8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:30:46.286158 7696 raft_consensus.cc:2454] T 1222b53d8c1a419ea185a04035531b2e P 80befb9ce7ad4b58b33f3f9fcd159f02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c73dc2975574aedbeb86ac18279ed45 in term 1.
I20260926 16:30:46.286666 7767 leader_election.cc:302] T 1222b53d8c1a419ea185a04035531b2e P 5c73dc2975574aedbeb86ac18279ed45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c73dc2975574aedbeb86ac18279ed45, 80befb9ce7ad4b58b33f3f9fcd159f02; no voters:
I20260926 16:30:46.286979 7912 raft_consensus.cc:2781] T 1222b53d8c1a419ea185a04035531b2e P 5c73dc2975574aedbeb86ac18279ed45 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:30:46.287072 7561 raft_consensus.cc:2454] T 1222b53d8c1a419ea185a04035531b2e P a7382ed5d79947958b4fef1f46e102d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c73dc2975574aedbeb86ac18279ed45 in term 1.
I20260926 16:30:46.287240 7912 raft_consensus.cc:686] T 1222b53d8c1a419ea185a04035531b2e P 5c73dc2975574aedbeb86ac18279ed45 [term 1 LEADER]: Becoming Leader. State: Replica: 5c73dc2975574aedbeb86ac18279ed45, State: Running, Role: LEADER
I20260926 16:30:46.287738 7912 consensus_queue.cc:237] T 1222b53d8c1a419ea185a04035531b2e P 5c73dc2975574aedbeb86ac18279ed45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7382ed5d79947958b4fef1f46e102d8" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 40727 } } peers { permanent_uuid: "80befb9ce7ad4b58b33f3f9fcd159f02" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36073 } } peers { permanent_uuid: "5c73dc2975574aedbeb86ac18279ed45" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 37879 } }
I20260926 16:30:46.289503 7914 tablet_replica.cc:439] T 1222b53d8c1a419ea185a04035531b2e P 5c73dc2975574aedbeb86ac18279ed45: TxnStatusTablet state changed. Reason: New leader 5c73dc2975574aedbeb86ac18279ed45. Latest consensus state: current_term: 1 leader_uuid: "5c73dc2975574aedbeb86ac18279ed45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7382ed5d79947958b4fef1f46e102d8" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 40727 } } peers { permanent_uuid: "80befb9ce7ad4b58b33f3f9fcd159f02" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36073 } } peers { permanent_uuid: "5c73dc2975574aedbeb86ac18279ed45" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 37879 } } }
I20260926 16:30:46.289834 7914 tablet_replica.cc:442] T 1222b53d8c1a419ea185a04035531b2e P 5c73dc2975574aedbeb86ac18279ed45: This TxnStatusTablet replica's current role is: LEADER
I20260926 16:30:46.290529 7944 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 16:30:46.290817 7944 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 16:30:46.291347 7399 catalog_manager.cc:5495] T 1222b53d8c1a419ea185a04035531b2e P 5c73dc2975574aedbeb86ac18279ed45 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c73dc2975574aedbeb86ac18279ed45 (127.6.33.3). New cstate: current_term: 1 leader_uuid: "5c73dc2975574aedbeb86ac18279ed45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7382ed5d79947958b4fef1f46e102d8" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 40727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80befb9ce7ad4b58b33f3f9fcd159f02" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c73dc2975574aedbeb86ac18279ed45" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 37879 } health_report { overall_health: HEALTHY } } }
I20260926 16:30:46.311172 7938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:46.306017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.3:37879 (local address 127.0.0.1:52820)
0926 16:30:46.306263 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0926 16:30:46.306289 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 16:30:46.306410 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:30:46.306916 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:30:46.306921 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:30:46.306935 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 16:30:46.307155 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:46.307164 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:46.308199 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:46.308206 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:46.309264 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:46.309275 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:46.310006 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:46.310014 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:46.310149 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:46.310203 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 16:30:46.310594 (+ 391us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 16:30:46.310617 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 16:30:46.310715 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 16:30:46.310868 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260926 16:30:46.311151 7902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:46.306396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52820 (local address 127.6.33.3:37879)
0926 16:30:46.306616 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:46.306622 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:46.306648 (+ 26us) server_negotiation.cc:407] Connection header received
0926 16:30:46.306705 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:46.306708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:46.306774 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:46.306886 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:30:46.307302 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:46.308072 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:46.309417 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:46.309826 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:46.309971 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:46.310396 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:30:46.310495 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:30:46.310591 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:46.310737 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 16:30:46.310813 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 16:30:46.315261 7696 raft_consensus.cc:1261] T 1222b53d8c1a419ea185a04035531b2e P 80befb9ce7ad4b58b33f3f9fcd159f02 [term 1 FOLLOWER]: Refusing update from remote peer 5c73dc2975574aedbeb86ac18279ed45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:30:46.315898 7561 raft_consensus.cc:1261] T 1222b53d8c1a419ea185a04035531b2e P a7382ed5d79947958b4fef1f46e102d8 [term 1 FOLLOWER]: Refusing update from remote peer 5c73dc2975574aedbeb86ac18279ed45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:30:46.316009 7914 consensus_queue.cc:1035] T 1222b53d8c1a419ea185a04035531b2e P 5c73dc2975574aedbeb86ac18279ed45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80befb9ce7ad4b58b33f3f9fcd159f02" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:30:46.316672 7914 consensus_queue.cc:1035] T 1222b53d8c1a419ea185a04035531b2e P 5c73dc2975574aedbeb86ac18279ed45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7382ed5d79947958b4fef1f46e102d8" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 40727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:30:46.318830 7911 tablet_replica.cc:439] T 1222b53d8c1a419ea185a04035531b2e P a7382ed5d79947958b4fef1f46e102d8: TxnStatusTablet state changed. Reason: New leader 5c73dc2975574aedbeb86ac18279ed45. Latest consensus state: current_term: 1 leader_uuid: "5c73dc2975574aedbeb86ac18279ed45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7382ed5d79947958b4fef1f46e102d8" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 40727 } } peers { permanent_uuid: "80befb9ce7ad4b58b33f3f9fcd159f02" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36073 } } peers { permanent_uuid: "5c73dc2975574aedbeb86ac18279ed45" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 37879 } } }
I20260926 16:30:46.319144 7911 tablet_replica.cc:442] T 1222b53d8c1a419ea185a04035531b2e P a7382ed5d79947958b4fef1f46e102d8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:30:46.320629 7912 tablet_replica.cc:439] T 1222b53d8c1a419ea185a04035531b2e P 5c73dc2975574aedbeb86ac18279ed45: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5c73dc2975574aedbeb86ac18279ed45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7382ed5d79947958b4fef1f46e102d8" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 40727 } } peers { permanent_uuid: "80befb9ce7ad4b58b33f3f9fcd159f02" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36073 } } peers { permanent_uuid: "5c73dc2975574aedbeb86ac18279ed45" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 37879 } } }
I20260926 16:30:46.320940 7912 tablet_replica.cc:442] T 1222b53d8c1a419ea185a04035531b2e P 5c73dc2975574aedbeb86ac18279ed45: This TxnStatusTablet replica's current role is: LEADER
I20260926 16:30:46.321528 7917 mvcc.cc:205] Tried to move back new op lower bound from 7333643248901165056 to 7333643248796921856. Current Snapshot: MvccSnapshot[applied={T|T < 7333643248901165056 or (T in {7333643248901165056})}]
I20260926 16:30:46.326953 7910 tablet_replica.cc:439] T 1222b53d8c1a419ea185a04035531b2e P 80befb9ce7ad4b58b33f3f9fcd159f02: TxnStatusTablet state changed. Reason: New leader 5c73dc2975574aedbeb86ac18279ed45. Latest consensus state: current_term: 1 leader_uuid: "5c73dc2975574aedbeb86ac18279ed45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7382ed5d79947958b4fef1f46e102d8" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 40727 } } peers { permanent_uuid: "80befb9ce7ad4b58b33f3f9fcd159f02" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36073 } } peers { permanent_uuid: "5c73dc2975574aedbeb86ac18279ed45" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 37879 } } }
I20260926 16:30:46.327225 7910 tablet_replica.cc:442] T 1222b53d8c1a419ea185a04035531b2e P 80befb9ce7ad4b58b33f3f9fcd159f02: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:30:46.329581 7914 tablet_replica.cc:439] T 1222b53d8c1a419ea185a04035531b2e P 5c73dc2975574aedbeb86ac18279ed45: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5c73dc2975574aedbeb86ac18279ed45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7382ed5d79947958b4fef1f46e102d8" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 40727 } } peers { permanent_uuid: "80befb9ce7ad4b58b33f3f9fcd159f02" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36073 } } peers { permanent_uuid: "5c73dc2975574aedbeb86ac18279ed45" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 37879 } } }
I20260926 16:30:46.329839 7914 tablet_replica.cc:442] T 1222b53d8c1a419ea185a04035531b2e P 5c73dc2975574aedbeb86ac18279ed45: This TxnStatusTablet replica's current role is: LEADER
I20260926 16:30:46.331514 7910 tablet_replica.cc:439] T 1222b53d8c1a419ea185a04035531b2e P 80befb9ce7ad4b58b33f3f9fcd159f02: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5c73dc2975574aedbeb86ac18279ed45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7382ed5d79947958b4fef1f46e102d8" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 40727 } } peers { permanent_uuid: "80befb9ce7ad4b58b33f3f9fcd159f02" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36073 } } peers { permanent_uuid: "5c73dc2975574aedbeb86ac18279ed45" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 37879 } } }
I20260926 16:30:46.331753 7910 tablet_replica.cc:442] T 1222b53d8c1a419ea185a04035531b2e P 80befb9ce7ad4b58b33f3f9fcd159f02: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:30:46.338393 7911 tablet_replica.cc:439] T 1222b53d8c1a419ea185a04035531b2e P a7382ed5d79947958b4fef1f46e102d8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5c73dc2975574aedbeb86ac18279ed45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7382ed5d79947958b4fef1f46e102d8" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 40727 } } peers { permanent_uuid: "80befb9ce7ad4b58b33f3f9fcd159f02" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36073 } } peers { permanent_uuid: "5c73dc2975574aedbeb86ac18279ed45" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 37879 } } }
I20260926 16:30:46.338685 7911 tablet_replica.cc:442] T 1222b53d8c1a419ea185a04035531b2e P a7382ed5d79947958b4fef1f46e102d8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:30:46.352743 7902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:46.346872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.3:33383 (local address 127.6.33.3:37879)
0926 16:30:46.347046 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:46.347050 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:46.347757 (+ 707us) server_negotiation.cc:407] Connection header received
0926 16:30:46.347934 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:46.347942 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:46.348017 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:46.348111 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:30:46.348611 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:46.349431 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:46.350900 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:46.351339 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:46.351478 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:46.351911 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:30:46.352066 (+ 155us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:30:46.352164 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:46.352303 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 16:30:46.352368 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 16:30:46.352813 7951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:46.346920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.3:37879 (local address 127.6.33.3:33383)
0926 16:30:46.347607 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0926 16:30:46.347622 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 16:30:46.347805 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:30:46.348133 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:30:46.348137 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:30:46.348146 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 16:30:46.348469 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:46.348478 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:46.349572 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:46.349577 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:46.350744 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:46.350755 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:46.351530 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:46.351538 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:46.351679 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:46.351734 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 16:30:46.352169 (+ 435us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 16:30:46.352181 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 16:30:46.352275 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 16:30:46.352407 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":396,"threads_started":1}
I20260926 16:30:46.362975 7561 raft_consensus.cc:1261] T f08ce60c785442b783f1af6519fb51c2 P a7382ed5d79947958b4fef1f46e102d8 [term 1 FOLLOWER]: Refusing update from remote peer 5c73dc2975574aedbeb86ac18279ed45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:30:46.363015 7696 raft_consensus.cc:1261] T f08ce60c785442b783f1af6519fb51c2 P 80befb9ce7ad4b58b33f3f9fcd159f02 [term 1 FOLLOWER]: Refusing update from remote peer 5c73dc2975574aedbeb86ac18279ed45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:30:46.363591 7912 consensus_queue.cc:1035] T f08ce60c785442b783f1af6519fb51c2 P 5c73dc2975574aedbeb86ac18279ed45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7382ed5d79947958b4fef1f46e102d8" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 40727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:30:46.363875 7948 consensus_queue.cc:1035] T f08ce60c785442b783f1af6519fb51c2 P 5c73dc2975574aedbeb86ac18279ed45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80befb9ce7ad4b58b33f3f9fcd159f02" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:30:46.371896 7919 mvcc.cc:205] Tried to move back new op lower bound from 7333643249097875456 to 7333643247943614464. Current Snapshot: MvccSnapshot[applied={T|T < 7333643249097875456}]
I20260926 16:30:46.439945 7398 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:30:46.440219 7398 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:30:46.443596 7398 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0922caa82bf5483b82a0c8cbd01a2f88: Sending DeleteTablet for 3 replicas of tablet f08ce60c785442b783f1af6519fb51c2
I20260926 16:30:46.444517 7537 tablet_service.cc:1514] Processing DeleteTablet for tablet f08ce60c785442b783f1af6519fb51c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:30:46 UTC) from {username='kudu', principal='kudu/127.6.33.62@KRBTEST.COM'} at 127.0.0.1:42952
I20260926 16:30:46.444518 7676 tablet_service.cc:1514] Processing DeleteTablet for tablet f08ce60c785442b783f1af6519fb51c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:30:46 UTC) from {username='kudu', principal='kudu/127.6.33.62@KRBTEST.COM'} at 127.0.0.1:34416
I20260926 16:30:46.444737 7814 tablet_service.cc:1514] Processing DeleteTablet for tablet f08ce60c785442b783f1af6519fb51c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:30:46 UTC) from {username='kudu', principal='kudu/127.6.33.62@KRBTEST.COM'} at 127.0.0.1:53188
I20260926 16:30:46.454998 7956 tablet_replica.cc:330] T f08ce60c785442b783f1af6519fb51c2 P 80befb9ce7ad4b58b33f3f9fcd159f02: stopping tablet replica
I20260926 16:30:46.455156 7955 tablet_replica.cc:330] T f08ce60c785442b783f1af6519fb51c2 P a7382ed5d79947958b4fef1f46e102d8: stopping tablet replica
I20260926 16:30:46.455267 7956 raft_consensus.cc:2229] T f08ce60c785442b783f1af6519fb51c2 P 80befb9ce7ad4b58b33f3f9fcd159f02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:30:46.455344 7955 raft_consensus.cc:2229] T f08ce60c785442b783f1af6519fb51c2 P a7382ed5d79947958b4fef1f46e102d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:30:46.455546 7956 raft_consensus.cc:2258] T f08ce60c785442b783f1af6519fb51c2 P 80befb9ce7ad4b58b33f3f9fcd159f02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:30:46.455546 7955 raft_consensus.cc:2258] T f08ce60c785442b783f1af6519fb51c2 P a7382ed5d79947958b4fef1f46e102d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:30:46.455963 7957 tablet_replica.cc:330] T f08ce60c785442b783f1af6519fb51c2 P 5c73dc2975574aedbeb86ac18279ed45: stopping tablet replica
I20260926 16:30:46.456162 7957 raft_consensus.cc:2229] T f08ce60c785442b783f1af6519fb51c2 P 5c73dc2975574aedbeb86ac18279ed45 [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:30:46.456604 7957 raft_consensus.cc:2258] T f08ce60c785442b783f1af6519fb51c2 P 5c73dc2975574aedbeb86ac18279ed45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:30:46.457801 7956 ts_tablet_manager.cc:1905] T f08ce60c785442b783f1af6519fb51c2 P 80befb9ce7ad4b58b33f3f9fcd159f02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:30:46.458050 7957 ts_tablet_manager.cc:1905] T f08ce60c785442b783f1af6519fb51c2 P 5c73dc2975574aedbeb86ac18279ed45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:30:46.456845 7955 ts_tablet_manager.cc:1905] T f08ce60c785442b783f1af6519fb51c2 P a7382ed5d79947958b4fef1f46e102d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:30:46.465471 7956 ts_tablet_manager.cc:1918] T f08ce60c785442b783f1af6519fb51c2 P 80befb9ce7ad4b58b33f3f9fcd159f02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 16:30:46.465607 7956 log.cc:1196] T f08ce60c785442b783f1af6519fb51c2 P 80befb9ce7ad4b58b33f3f9fcd159f02: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/ts-1/wal/wals/f08ce60c785442b783f1af6519fb51c2
I20260926 16:30:46.466055 7956 ts_tablet_manager.cc:1939] T f08ce60c785442b783f1af6519fb51c2 P 80befb9ce7ad4b58b33f3f9fcd159f02: Deleting consensus metadata
I20260926 16:30:46.467202 7385 catalog_manager.cc:4841] TS 80befb9ce7ad4b58b33f3f9fcd159f02 (127.6.33.2:36073): tablet f08ce60c785442b783f1af6519fb51c2 (table test-table [id=3e626a31f66645dda0b7ddf3ab4d2261]) successfully deleted
I20260926 16:30:46.468333 7957 ts_tablet_manager.cc:1918] T f08ce60c785442b783f1af6519fb51c2 P 5c73dc2975574aedbeb86ac18279ed45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 16:30:46.468464 7957 log.cc:1196] T f08ce60c785442b783f1af6519fb51c2 P 5c73dc2975574aedbeb86ac18279ed45: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/ts-2/wal/wals/f08ce60c785442b783f1af6519fb51c2
I20260926 16:30:46.468956 7957 ts_tablet_manager.cc:1939] T f08ce60c785442b783f1af6519fb51c2 P 5c73dc2975574aedbeb86ac18279ed45: Deleting consensus metadata
I20260926 16:30:46.470137 7384 catalog_manager.cc:4841] TS 5c73dc2975574aedbeb86ac18279ed45 (127.6.33.3:37879): tablet f08ce60c785442b783f1af6519fb51c2 (table test-table [id=3e626a31f66645dda0b7ddf3ab4d2261]) successfully deleted
I20260926 16:30:46.472462 7955 ts_tablet_manager.cc:1918] T f08ce60c785442b783f1af6519fb51c2 P a7382ed5d79947958b4fef1f46e102d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 16:30:46.472613 7955 log.cc:1196] T f08ce60c785442b783f1af6519fb51c2 P a7382ed5d79947958b4fef1f46e102d8: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440233850748-6276-0/minicluster-data/ts-0/wal/wals/f08ce60c785442b783f1af6519fb51c2
I20260926 16:30:46.473076 7955 ts_tablet_manager.cc:1939] T f08ce60c785442b783f1af6519fb51c2 P a7382ed5d79947958b4fef1f46e102d8: Deleting consensus metadata
I20260926 16:30:46.474124 7387 catalog_manager.cc:4841] TS a7382ed5d79947958b4fef1f46e102d8 (127.6.33.1:40727): tablet f08ce60c785442b783f1af6519fb51c2 (table test-table [id=3e626a31f66645dda0b7ddf3ab4d2261]) successfully deleted
Sep 26 16:30:46 dist-test-slave-khd3 krb5kdc[7355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440245, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.33.1@KRBTEST.COM
I20260926 16:30:46.488031 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:46.460440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56376 (local address 127.6.33.1:40727)
0926 16:30:46.471529 (+ 11089us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:46.471536 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:46.471555 (+ 19us) server_negotiation.cc:407] Connection header received
0926 16:30:46.471608 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:46.471613 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:46.471708 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:46.471814 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:30:46.473690 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:46.474765 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:46.476624 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:46.477227 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:46.477402 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:46.482157 (+ 4755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:30:46.482171 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:30:46.482187 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:30:46.482223 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:30:46.485186 (+ 2963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:46.485893 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:46.485900 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:46.485903 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:46.486058 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:46.486446 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:46.486451 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:46.486453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:46.486845 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:30:46.487148 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:46.487457 (+ 309us) server_negotiation.cc:299] Negotiation successful
0926 16:30:46.487682 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10952,"thread_start_us":10841,"threads_started":1}
W20260926 16:30:46.488749 7602 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56376
I20260926 16:30:46.506026 7939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:46.491669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40030 (local address 127.6.33.62:39505)
0926 16:30:46.491872 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:46.491880 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:46.492064 (+ 184us) server_negotiation.cc:407] Connection header received
0926 16:30:46.492226 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:46.492232 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:46.492316 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:46.492421 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:30:46.493746 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:46.494904 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:46.496165 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:46.496903 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:46.497042 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:46.499567 (+ 2525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:30:46.499575 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:30:46.499580 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:30:46.499626 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:30:46.503018 (+ 3392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:46.503874 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:46.503879 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:46.503881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:46.504007 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:46.504418 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:46.504465 (+ 47us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:46.504468 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:46.504872 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:30:46.505140 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:46.505468 (+ 328us) server_negotiation.cc:299] Negotiation successful
0926 16:30:46.505703 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260926 16:30:46.509212 7398 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40030
I20260926 16:30:46.511116 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 7471
I20260926 16:30:46.532953 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:46.513462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56392 (local address 127.6.33.1:40727)
0926 16:30:46.513690 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:46.513696 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:46.517302 (+ 3606us) server_negotiation.cc:407] Connection header received
0926 16:30:46.517448 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:46.517454 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:46.517564 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:46.517678 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:30:46.519209 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:46.520429 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:46.522356 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:46.523055 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:46.523170 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:46.526161 (+ 2991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:30:46.526168 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:30:46.526173 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:30:46.526212 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:30:46.529855 (+ 3643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:46.530729 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:46.530734 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:46.530738 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:46.530886 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:46.531261 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:46.531266 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:46.531270 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:46.531585 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:30:46.531841 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:46.532366 (+ 525us) server_negotiation.cc:299] Negotiation successful
0926 16:30:46.532589 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260926 16:30:46.534004 7602 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56392
W20260926 16:30:46.534592 6276 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 7471: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 16:30:46.534761 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 7471
I20260926 16:30:46.551865 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 7611
Sep 26 16:30:46 dist-test-slave-khd3 krb5kdc[7355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440245, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.33.2@KRBTEST.COM
I20260926 16:30:46.568714 7964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:46.552732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58954 (local address 127.6.33.2:36073)
0926 16:30:46.553368 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:46.553374 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:46.553396 (+ 22us) server_negotiation.cc:407] Connection header received
0926 16:30:46.553462 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:46.553468 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:46.553558 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:46.553677 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:30:46.554880 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:46.556076 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:46.557487 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:46.558211 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:46.558363 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:46.562549 (+ 4186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:30:46.562563 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:30:46.562577 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:30:46.562614 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:30:46.565600 (+ 2986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:46.566303 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:46.566312 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:46.566314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:46.566453 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:46.567195 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:46.567198 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:46.567200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:46.567536 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:30:46.567824 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:46.568117 (+ 293us) server_negotiation.cc:299] Negotiation successful
0926 16:30:46.568374 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":288,"threads_started":1}
W20260926 16:30:46.569433 7736 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58954
W20260926 16:30:46.570115 6276 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 7611: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 16:30:46.570298 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 7611
I20260926 16:30:46.588589 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 7750
Sep 26 16:30:46 dist-test-slave-khd3 krb5kdc[7355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440245, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.33.3@KRBTEST.COM
I20260926 16:30:46.604487 7902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:46.589477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52836 (local address 127.6.33.3:37879)
0926 16:30:46.589678 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:46.589695 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:46.589750 (+ 55us) server_negotiation.cc:407] Connection header received
0926 16:30:46.589959 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:46.589964 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:46.590032 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:46.590142 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:30:46.591365 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:46.592140 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:46.593517 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:46.594001 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:46.594135 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:46.598280 (+ 4145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:30:46.598292 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:30:46.598309 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:30:46.598346 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:30:46.601572 (+ 3226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:46.602289 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:46.602297 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:46.602300 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:46.602426 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:46.602812 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:46.602818 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:46.602821 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:46.603235 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:30:46.603595 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:46.603865 (+ 270us) server_negotiation.cc:299] Negotiation successful
0926 16:30:46.604103 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260926 16:30:46.605276 7880 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52836
W20260926 16:30:46.605986 6276 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 7750: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 16:30:46.606142 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 7750
I20260926 16:30:46.629745 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 7368
I20260926 16:30:46.630885 7440 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:30:46.746680 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 7368
2026-09-26T16:30:46Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4582 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:30:46 dist-test-slave-khd3 krb5kdc[7968](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:30:46 dist-test-slave-khd3 krb5kdc[7968](info): set up 2 sockets
Sep 26 16:30:46 dist-test-slave-khd3 krb5kdc[7968](info): commencing operation
krb5kdc: starting...
W20260926 16:30:47.837930 6276 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Sep 26 16:30:47 dist-test-slave-khd3 krb5kdc[7968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:30:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:30:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab.
Entry for principal kudu/127.6.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab.
I20260926 16:30:48.033547 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:41329
--webserver_interface=127.6.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:44433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.33.62:41329
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:30:48.363479 7981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:30:48.363842 7981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:30:48.364012 7981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:30:48.371413 7981 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:30:48.371587 7981 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 16:30:48.371690 7981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:30:48.371814 7981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:30:48.371899 7981 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:30:48.371989 7981 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:30:48.382205 7981 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:44433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.33.62:41329
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:41329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:30:48.383544 7981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:30:48.384980 7981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:30:48.397807 7986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:30:48.398168 7989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:30:48.397783 7987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:30:48.398682 7981 server_base.cc:1034] running on GCE node
I20260926 16:30:48.399566 7981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:30:48.401198 7981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:30:48.402441 7981 hybrid_clock.cc:648] HybridClock initialized: now 1790440248402353 us; error 71 us; skew 500 ppm
I20260926 16:30:48.402834 7981 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:30:48 dist-test-slave-khd3 krb5kdc[7968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440248, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:30:48.413089 7981 init.cc:377] Logged in from keytab as kudu/127.6.33.62@KRBTEST.COM (short username kudu)
I20260926 16:30:48.414811 7981 webserver.cc:460] Webserver started at http://127.6.33.62:41269/ using document root <none> and password file <none>
I20260926 16:30:48.415632 7981 fs_manager.cc:359] Metadata directory not provided
I20260926 16:30:48.415730 7981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:30:48.416102 7981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:30:48.419489 7981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/master-0/data/instance:
uuid: "4ed326500513428ebd1beeff75c87965"
format_stamp: "Formatted at 2026-09-26 16:30:48 on dist-test-slave-khd3"
server_key: "363bca75fbb8948f9628be4ea09b0663"
server_key_iv: "162e89e1d529b28ac8845de6bb441948"
server_key_version: "encryptionkey@0"
I20260926 16:30:48.420372 7981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/master-0/wal/instance:
uuid: "4ed326500513428ebd1beeff75c87965"
format_stamp: "Formatted at 2026-09-26 16:30:48 on dist-test-slave-khd3"
server_key: "363bca75fbb8948f9628be4ea09b0663"
server_key_iv: "162e89e1d529b28ac8845de6bb441948"
server_key_version: "encryptionkey@0"
I20260926 16:30:48.426271 7981 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 16:30:48.430306 7996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:30:48.432042 7981 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:30:48.432205 7981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/master-0/wal
uuid: "4ed326500513428ebd1beeff75c87965"
format_stamp: "Formatted at 2026-09-26 16:30:48 on dist-test-slave-khd3"
server_key: "363bca75fbb8948f9628be4ea09b0663"
server_key_iv: "162e89e1d529b28ac8845de6bb441948"
server_key_version: "encryptionkey@0"
I20260926 16:30:48.432411 7981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:30:48.460202 7981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:30:48.465157 7981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:30:48.465412 7981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:30:48.487785 7981 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.62:41329
I20260926 16:30:48.487828 8057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.62:41329 every 8 connection(s)
I20260926 16:30:48.489248 7981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
I20260926 16:30:48.492537 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 7981
I20260926 16:30:48.492851 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/master-0/wal/instance
I20260926 16:30:48.493490 6276 external_mini_cluster.cc:1428] Setting key 1c11e05fd192bea5bc0294648ab12c49
I20260926 16:30:48.493604 8058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:30:48.509186 8058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ed326500513428ebd1beeff75c87965: Bootstrap starting.
Sep 26 16:30:48 dist-test-slave-khd3 krb5kdc[7968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:30:48.513522 8058 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ed326500513428ebd1beeff75c87965: Neither blocks nor log segments found. Creating new log.
I20260926 16:30:48.514971 8058 log.cc:824] T 00000000000000000000000000000000 P 4ed326500513428ebd1beeff75c87965: Log is configured to *not* fsync() on all Append() calls
I20260926 16:30:48.518802 8058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ed326500513428ebd1beeff75c87965: No bootstrap required, opened a new log
I20260926 16:30:48.520308 8061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:48.495619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52200 (local address 127.6.33.62:41329)
0926 16:30:48.496486 (+ 867us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:48.496501 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:48.496543 (+ 42us) server_negotiation.cc:407] Connection header received
0926 16:30:48.498558 (+ 2015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:48.498583 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:48.499012 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:48.499394 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:30:48.501346 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:48.502728 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:48.504536 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:48.505432 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:48.505614 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:48.510799 (+ 5185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:30:48.510821 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:30:48.510847 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:30:48.510919 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:30:48.515238 (+ 4319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:48.516332 (+ 1094us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:48.516342 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:48.516349 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:48.516577 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:48.517024 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:48.517029 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:48.517031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:48.517717 (+ 686us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:30:48.518039 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:48.518443 (+ 404us) server_negotiation.cc:299] Negotiation successful
0926 16:30:48.518912 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":378,"threads_started":1}
I20260926 16:30:48.527469 8058 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ed326500513428ebd1beeff75c87965 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ed326500513428ebd1beeff75c87965" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 41329 } }
I20260926 16:30:48.527910 8058 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ed326500513428ebd1beeff75c87965 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:30:48.528019 8058 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ed326500513428ebd1beeff75c87965 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ed326500513428ebd1beeff75c87965, State: Initialized, Role: FOLLOWER
I20260926 16:30:48.528692 8058 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ed326500513428ebd1beeff75c87965 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ed326500513428ebd1beeff75c87965" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 41329 } }
I20260926 16:30:48.528977 8058 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ed326500513428ebd1beeff75c87965 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:30:48.529109 8058 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ed326500513428ebd1beeff75c87965 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:30:48.529282 8058 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ed326500513428ebd1beeff75c87965 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:30:48.531724 8058 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ed326500513428ebd1beeff75c87965 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ed326500513428ebd1beeff75c87965" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 41329 } }
I20260926 16:30:48.532262 8058 leader_election.cc:302] T 00000000000000000000000000000000 P 4ed326500513428ebd1beeff75c87965 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ed326500513428ebd1beeff75c87965; no voters:
I20260926 16:30:48.532929 8058 leader_election.cc:288] T 00000000000000000000000000000000 P 4ed326500513428ebd1beeff75c87965 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:30:48.533069 8063 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ed326500513428ebd1beeff75c87965 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:30:48.533820 8063 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ed326500513428ebd1beeff75c87965 [term 1 LEADER]: Becoming Leader. State: Replica: 4ed326500513428ebd1beeff75c87965, State: Running, Role: LEADER
I20260926 16:30:48.534376 8063 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ed326500513428ebd1beeff75c87965 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ed326500513428ebd1beeff75c87965" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 41329 } }
I20260926 16:30:48.535121 8058 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ed326500513428ebd1beeff75c87965 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:30:48.538134 8064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ed326500513428ebd1beeff75c87965 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ed326500513428ebd1beeff75c87965" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ed326500513428ebd1beeff75c87965" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 41329 } } }
I20260926 16:30:48.538592 8064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ed326500513428ebd1beeff75c87965 [sys.catalog]: This master's current role is: LEADER
I20260926 16:30:48.538880 8065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ed326500513428ebd1beeff75c87965 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ed326500513428ebd1beeff75c87965. Latest consensus state: current_term: 1 leader_uuid: "4ed326500513428ebd1beeff75c87965" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ed326500513428ebd1beeff75c87965" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 41329 } } }
I20260926 16:30:48.539211 8065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ed326500513428ebd1beeff75c87965 [sys.catalog]: This master's current role is: LEADER
I20260926 16:30:48.545526 8072 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:30:48.550132 8072 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:30:48.558823 8072 catalog_manager.cc:1347] Generated new cluster ID: 807fe6ada1cd4e2b94a9dc7da28ee89f
I20260926 16:30:48.558948 8072 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:30:48.567934 8072 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:30:48.569471 8072 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:30:48.582820 8072 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ed326500513428ebd1beeff75c87965: Generated new TSK 0
I20260926 16:30:48.583732 8072 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.33.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab.
Entry for principal kudu/127.6.33.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab.
I20260926 16:30:48.632537 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.1:0
--local_ip_for_outbound_sockets=127.6.33.1
--webserver_interface=127.6.33.1
--webserver_port=0
--tserver_master_addrs=127.6.33.62:41329
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab
--principal=kudu/127.6.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:44433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:30:48.967665 8084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:30:48.967948 8084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:30:48.968116 8084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:30:48.975564 8084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:30:48.975682 8084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:30:48.975832 8084 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 16:30:48.975891 8084 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 16:30:48.976023 8084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.1
I20260926 16:30:48.986285 8084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:44433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab
--principal=kudu/127.6.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.33.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.33.62:41329
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.1
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:30:48.987641 8084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:30:48.989141 8084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:30:49.003064 8092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:30:49.003064 8090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:30:49.003453 8089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:30:49.003947 8084 server_base.cc:1034] running on GCE node
I20260926 16:30:49.004619 8084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:30:49.005968 8084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:30:49.007246 8084 hybrid_clock.cc:648] HybridClock initialized: now 1790440249007152 us; error 54 us; skew 500 ppm
I20260926 16:30:49.007637 8084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:30:49 dist-test-slave-khd3 krb5kdc[7968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440249, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:30:49.011823 8084 init.cc:377] Logged in from keytab as kudu/127.6.33.1@KRBTEST.COM (short username kudu)
I20260926 16:30:49.013856 8084 webserver.cc:460] Webserver started at http://127.6.33.1:38701/ using document root <none> and password file <none>
I20260926 16:30:49.014799 8084 fs_manager.cc:359] Metadata directory not provided
I20260926 16:30:49.014966 8084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:30:49.015316 8084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:30:49.018501 8084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/ts-0/data/instance:
uuid: "9cf28505064b4b4ba7581338c203ee66"
format_stamp: "Formatted at 2026-09-26 16:30:49 on dist-test-slave-khd3"
server_key: "a1677f64f914c8ab45d44f462d579840"
server_key_iv: "939f21fce0fd35160720220146d90c5b"
server_key_version: "encryptionkey@0"
I20260926 16:30:49.019282 8084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/ts-0/wal/instance:
uuid: "9cf28505064b4b4ba7581338c203ee66"
format_stamp: "Formatted at 2026-09-26 16:30:49 on dist-test-slave-khd3"
server_key: "a1677f64f914c8ab45d44f462d579840"
server_key_iv: "939f21fce0fd35160720220146d90c5b"
server_key_version: "encryptionkey@0"
I20260926 16:30:49.024734 8084 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 16:30:49.028415 8099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:30:49.029968 8084 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 16:30:49.030161 8084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/ts-0/wal
uuid: "9cf28505064b4b4ba7581338c203ee66"
format_stamp: "Formatted at 2026-09-26 16:30:49 on dist-test-slave-khd3"
server_key: "a1677f64f914c8ab45d44f462d579840"
server_key_iv: "939f21fce0fd35160720220146d90c5b"
server_key_version: "encryptionkey@0"
I20260926 16:30:49.030365 8084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:30:49.045344 8084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:30:49.049876 8084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:30:49.050179 8084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:30:49.052870 8084 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:30:49.053005 8084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:30:49.053144 8084 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:30:49.053241 8084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 16:30:49 dist-test-slave-khd3 krb5kdc[7968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440249, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.1@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:30:49.081594 8108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:49.054000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:36889 (local address 127.6.33.62:41329)
0926 16:30:49.054437 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:49.054441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:49.056010 (+ 1569us) server_negotiation.cc:407] Connection header received
0926 16:30:49.058177 (+ 2167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:49.058184 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:49.058282 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:49.058408 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:30:49.061692 (+ 3284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:49.062618 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:49.066201 (+ 3583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:49.066801 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:49.066959 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:49.073330 (+ 6371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:30:49.073340 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:30:49.073347 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:30:49.073403 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:30:49.077348 (+ 3945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:49.078733 (+ 1385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:49.078739 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:49.078744 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:49.078915 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:49.079535 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:49.079541 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:49.079545 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:49.079875 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:30:49.080051 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:49.080908 (+ 857us) server_negotiation.cc:299] Negotiation successful
0926 16:30:49.081171 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":245,"threads_started":1}
I20260926 16:30:49.083158 8109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:49.054760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:41329 (local address 127.6.33.1:36889)
0926 16:30:49.055730 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0926 16:30:49.055778 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 16:30:49.057848 (+ 2070us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:30:49.059007 (+ 1159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:30:49.059024 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:30:49.060170 (+ 1146us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:30:49.061396 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:49.061432 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:49.062817 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:49.062829 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:49.065892 (+ 3063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:49.065931 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:49.067295 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:49.067306 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:49.067631 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:49.068659 (+ 1028us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:30:49.068694 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:30:49.072913 (+ 4219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:30:49.077560 (+ 4647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:30:49.077571 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:30:49.077599 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:30:49.078487 (+ 888us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:30:49.079054 (+ 567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:30:49.079062 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:30:49.079069 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:30:49.079372 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:30:49.080027 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:30:49.080036 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:30:49.080441 (+ 405us) client_negotiation.cc:769] Sending connection context
0926 16:30:49.080794 (+ 353us) client_negotiation.cc:240] Negotiation successful
0926 16:30:49.081405 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":413,"threads_started":1}
I20260926 16:30:49.106590 8182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:49.090972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:41329 (local address 127.6.33.1:33057)
0926 16:30:49.091903 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0926 16:30:49.091929 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 16:30:49.092110 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:30:49.092465 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:30:49.092472 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:30:49.093244 (+ 772us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:30:49.093804 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:49.093825 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:49.095072 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:49.095081 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:49.097118 (+ 2037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:49.097138 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:49.097981 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:49.097989 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:49.098139 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:49.098969 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:30:49.099002 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:30:49.100355 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:30:49.103971 (+ 3616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:30:49.103977 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:30:49.103981 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:30:49.104357 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:30:49.104730 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:30:49.104736 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:30:49.104740 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:30:49.104894 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:30:49.105501 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:30:49.105506 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:30:49.105727 (+ 221us) client_negotiation.cc:769] Sending connection context
0926 16:30:49.105947 (+ 220us) client_negotiation.cc:240] Negotiation successful
0926 16:30:49.106254 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"thread_start_us":511,"threads_started":1}
I20260926 16:30:49.106923 8108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:49.091026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:33057 (local address 127.6.33.62:41329)
0926 16:30:49.091254 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:49.091265 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:49.092032 (+ 767us) server_negotiation.cc:407] Connection header received
0926 16:30:49.092241 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:49.092245 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:49.092311 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:49.092399 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:30:49.093977 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:49.094926 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:49.097337 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:49.097747 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:49.097887 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:49.100555 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:30:49.100560 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:30:49.100563 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:30:49.100594 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:30:49.103833 (+ 3239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:49.104484 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:49.104489 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:49.104492 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:49.104625 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:49.105001 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:49.105008 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:49.105012 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:49.105399 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:30:49.105540 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:49.106123 (+ 583us) server_negotiation.cc:299] Negotiation successful
0926 16:30:49.106424 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 16:30:49.112653 8084 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.1:38069
I20260926 16:30:49.112666 8219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.1:38069 every 8 connection(s)
I20260926 16:30:49.114045 8084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
I20260926 16:30:49.116565 8220 heartbeater.cc:347] Connected to a master server at 127.6.33.62:41329
I20260926 16:30:49.116982 8220 heartbeater.cc:464] Registering TS with master...
I20260926 16:30:49.117759 8220 heartbeater.cc:511] Master 127.6.33.62:41329 requested a full tablet report, sending...
I20260926 16:30:49.120012 8013 ts_manager.cc:194] Registered new tserver with Master: 9cf28505064b4b4ba7581338c203ee66 (127.6.33.1:38069)
I20260926 16:30:49.121594 8013 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.33.1@KRBTEST.COM'} at 127.6.33.1:36889
I20260926 16:30:49.122232 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 8084
I20260926 16:30:49.122469 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/ts-0/wal/instance
I20260926 16:30:49.123229 6276 external_mini_cluster.cc:1428] Setting key 8b4d554ed33ee2816ffe656c077db26a
WARNING: no policy specified for kudu/127.6.33.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab.
Entry for principal kudu/127.6.33.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab.
I20260926 16:30:49.164587 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.2:0
--local_ip_for_outbound_sockets=127.6.33.2
--webserver_interface=127.6.33.2
--webserver_port=0
--tserver_master_addrs=127.6.33.62:41329
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab
--principal=kudu/127.6.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:44433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:30:49.493077 8224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:30:49.493340 8224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:30:49.493498 8224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:30:49.500943 8224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:30:49.501053 8224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:30:49.501231 8224 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 16:30:49.501312 8224 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 16:30:49.501420 8224 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.2
I20260926 16:30:49.511919 8224 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:44433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab
--principal=kudu/127.6.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.33.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.33.62:41329
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.2
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:30:49.513327 8224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:30:49.514624 8224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:30:49.528137 8230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:30:49.528358 8229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:30:49.528143 8232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:30:49.529249 8224 server_base.cc:1034] running on GCE node
I20260926 16:30:49.529976 8224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:30:49.531060 8224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:30:49.532311 8224 hybrid_clock.cc:648] HybridClock initialized: now 1790440249532278 us; error 58 us; skew 500 ppm
I20260926 16:30:49.532645 8224 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:30:49 dist-test-slave-khd3 krb5kdc[7968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440249, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:30:49.536157 8224 init.cc:377] Logged in from keytab as kudu/127.6.33.2@KRBTEST.COM (short username kudu)
I20260926 16:30:49.538019 8224 webserver.cc:460] Webserver started at http://127.6.33.2:36713/ using document root <none> and password file <none>
I20260926 16:30:49.538862 8224 fs_manager.cc:359] Metadata directory not provided
I20260926 16:30:49.539018 8224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:30:49.539379 8224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:30:49.542620 8224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/ts-1/data/instance:
uuid: "6a4cdd1c228144b1a0a7bd99cab216cf"
format_stamp: "Formatted at 2026-09-26 16:30:49 on dist-test-slave-khd3"
server_key: "f24c5ea1daee287504fc722c638b77be"
server_key_iv: "177c5d1e688014759b32fe58bf6c323c"
server_key_version: "encryptionkey@0"
I20260926 16:30:49.543475 8224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/ts-1/wal/instance:
uuid: "6a4cdd1c228144b1a0a7bd99cab216cf"
format_stamp: "Formatted at 2026-09-26 16:30:49 on dist-test-slave-khd3"
server_key: "f24c5ea1daee287504fc722c638b77be"
server_key_iv: "177c5d1e688014759b32fe58bf6c323c"
server_key_version: "encryptionkey@0"
I20260926 16:30:49.549218 8224 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260926 16:30:49.553098 8239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:30:49.554560 8224 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:30:49.554802 8224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/ts-1/wal
uuid: "6a4cdd1c228144b1a0a7bd99cab216cf"
format_stamp: "Formatted at 2026-09-26 16:30:49 on dist-test-slave-khd3"
server_key: "f24c5ea1daee287504fc722c638b77be"
server_key_iv: "177c5d1e688014759b32fe58bf6c323c"
server_key_version: "encryptionkey@0"
I20260926 16:30:49.555032 8224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:30:49.568418 8224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:30:49.573174 8224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:30:49.573475 8224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:30:49.576107 8224 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:30:49.576247 8224 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:30:49.576413 8224 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:30:49.576505 8224 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 16:30:49 dist-test-slave-khd3 krb5kdc[7968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440249, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.2@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:30:49.604709 8248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:49.577662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:41329 (local address 127.6.33.2:46979)
0926 16:30:49.578565 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0926 16:30:49.578650 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0926 16:30:49.581636 (+ 2986us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:30:49.582637 (+ 1001us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:30:49.582651 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:30:49.583546 (+ 895us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:30:49.584741 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:49.584798 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:49.586149 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:49.586155 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:49.589101 (+ 2946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:49.589128 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:49.590161 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:49.590175 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:49.590552 (+ 377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:49.591443 (+ 891us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:30:49.591476 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:30:49.595752 (+ 4276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:30:49.599991 (+ 4239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:30:49.600001 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:30:49.600020 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:30:49.600615 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:30:49.601142 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:30:49.601146 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:30:49.601148 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:30:49.601357 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:30:49.602081 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:30:49.602089 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:30:49.602498 (+ 409us) client_negotiation.cc:769] Sending connection context
0926 16:30:49.602914 (+ 416us) client_negotiation.cc:240] Negotiation successful
0926 16:30:49.603444 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":348,"threads_started":1}
I20260926 16:30:49.605556 8108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:49.577089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:46979 (local address 127.6.33.62:41329)
0926 16:30:49.577277 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:49.577284 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:49.578817 (+ 1533us) server_negotiation.cc:407] Connection header received
0926 16:30:49.582045 (+ 3228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:49.582054 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:49.582154 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:49.582302 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:30:49.584978 (+ 2676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:49.585948 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:49.589300 (+ 3352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:49.589884 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:49.590033 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:49.596103 (+ 6070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:30:49.596111 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:30:49.596118 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:30:49.596175 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:30:49.599836 (+ 3661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:49.600821 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:49.600830 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:49.600834 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:49.601019 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:49.601493 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:49.601499 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:49.601504 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:49.601932 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:30:49.602096 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:49.604998 (+ 2902us) server_negotiation.cc:299] Negotiation successful
0926 16:30:49.605252 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 16:30:49.621523 8224 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.2:37269
I20260926 16:30:49.621556 8358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.2:37269 every 8 connection(s)
I20260926 16:30:49.623057 8224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
I20260926 16:30:49.624166 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 8224
I20260926 16:30:49.624367 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/ts-1/wal/instance
I20260926 16:30:49.625029 6276 external_mini_cluster.cc:1428] Setting key d866748bf0c4025f2ed6580649a15d94
I20260926 16:30:49.626255 8359 heartbeater.cc:347] Connected to a master server at 127.6.33.62:41329
I20260926 16:30:49.626606 8359 heartbeater.cc:464] Registering TS with master...
I20260926 16:30:49.627564 8359 heartbeater.cc:511] Master 127.6.33.62:41329 requested a full tablet report, sending...
I20260926 16:30:49.629213 8013 ts_manager.cc:194] Registered new tserver with Master: 6a4cdd1c228144b1a0a7bd99cab216cf (127.6.33.2:37269)
I20260926 16:30:49.630476 8013 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.33.2@KRBTEST.COM'} at 127.6.33.2:46979
I20260926 16:30:49.634102 8335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:49.611738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:41329 (local address 127.6.33.2:50397)
0926 16:30:49.612594 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0926 16:30:49.612611 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 16:30:49.612924 (+ 313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:30:49.613471 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:30:49.613475 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:30:49.614179 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:30:49.614743 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:49.614766 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:49.616572 (+ 1806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:49.616576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:49.618048 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:49.618065 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:49.619263 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:49.619269 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:49.619439 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:49.620220 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:30:49.620251 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:30:49.621277 (+ 1026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:30:49.630904 (+ 9627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:30:49.630913 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:30:49.630920 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:30:49.631568 (+ 648us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:30:49.632072 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:30:49.632078 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:30:49.632088 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:30:49.632247 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:30:49.632994 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:30:49.632999 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:30:49.633176 (+ 177us) client_negotiation.cc:769] Sending connection context
0926 16:30:49.633301 (+ 125us) client_negotiation.cc:240] Negotiation successful
0926 16:30:49.633632 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"thread_start_us":470,"threads_started":1}
I20260926 16:30:49.634997 8108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:49.611738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:50397 (local address 127.6.33.62:41329)
0926 16:30:49.611936 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:49.611944 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:49.612736 (+ 792us) server_negotiation.cc:407] Connection header received
0926 16:30:49.613115 (+ 379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:49.613124 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:49.613250 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:49.613401 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:30:49.614967 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:49.616401 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:49.618328 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:49.619026 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:49.619208 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:49.621502 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:30:49.621507 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:30:49.621510 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:30:49.621553 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:30:49.630734 (+ 9181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:49.631734 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:49.631739 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:49.631743 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:49.631904 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:49.632396 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:49.632401 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:49.632405 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:49.632859 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:30:49.634383 (+ 1524us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:49.634518 (+ 135us) server_negotiation.cc:299] Negotiation successful
0926 16:30:49.634744 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
WARNING: no policy specified for kudu/127.6.33.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab.
Entry for principal kudu/127.6.33.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab.
I20260926 16:30:49.673532 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.3:0
--local_ip_for_outbound_sockets=127.6.33.3
--webserver_interface=127.6.33.3
--webserver_port=0
--tserver_master_addrs=127.6.33.62:41329
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab
--principal=kudu/127.6.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:44433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:30:50.016132 8363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:30:50.016453 8363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:30:50.016557 8363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:30:50.024129 8363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:30:50.024267 8363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:30:50.024353 8363 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 16:30:50.024405 8363 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 16:30:50.024467 8363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.3
I20260926 16:30:50.034969 8363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:44433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab
--principal=kudu/127.6.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.33.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.33.62:41329
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.3
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:30:50.036633 8363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:30:50.038123 8363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:30:50.052739 8368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:30:50.052788 8369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:30:50.052994 8371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:30:50.054343 8363 server_base.cc:1034] running on GCE node
I20260926 16:30:50.055154 8363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:30:50.056275 8363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:30:50.057549 8363 hybrid_clock.cc:648] HybridClock initialized: now 1790440250057515 us; error 35 us; skew 500 ppm
I20260926 16:30:50.057998 8363 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:30:50 dist-test-slave-khd3 krb5kdc[7968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440250, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:30:50.062057 8363 init.cc:377] Logged in from keytab as kudu/127.6.33.3@KRBTEST.COM (short username kudu)
I20260926 16:30:50.063673 8363 webserver.cc:460] Webserver started at http://127.6.33.3:33011/ using document root <none> and password file <none>
I20260926 16:30:50.064450 8363 fs_manager.cc:359] Metadata directory not provided
I20260926 16:30:50.064607 8363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:30:50.065057 8363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:30:50.068409 8363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/ts-2/data/instance:
uuid: "49a7b8cf178d46f28c018c0bda345304"
format_stamp: "Formatted at 2026-09-26 16:30:50 on dist-test-slave-khd3"
server_key: "88fd2661b150ac986a690825cdbc7053"
server_key_iv: "afa849ef5f5434b8bf9910d2d749dd77"
server_key_version: "encryptionkey@0"
I20260926 16:30:50.069289 8363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/ts-2/wal/instance:
uuid: "49a7b8cf178d46f28c018c0bda345304"
format_stamp: "Formatted at 2026-09-26 16:30:50 on dist-test-slave-khd3"
server_key: "88fd2661b150ac986a690825cdbc7053"
server_key_iv: "afa849ef5f5434b8bf9910d2d749dd77"
server_key_version: "encryptionkey@0"
I20260926 16:30:50.075237 8363 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 16:30:50.079217 8378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:30:50.080713 8363 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 16:30:50.080979 8363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/ts-2/wal
uuid: "49a7b8cf178d46f28c018c0bda345304"
format_stamp: "Formatted at 2026-09-26 16:30:50 on dist-test-slave-khd3"
server_key: "88fd2661b150ac986a690825cdbc7053"
server_key_iv: "afa849ef5f5434b8bf9910d2d749dd77"
server_key_version: "encryptionkey@0"
I20260926 16:30:50.081187 8363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:30:50.103014 8363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:30:50.107986 8363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:30:50.108289 8363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:30:50.111142 8363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:30:50.111244 8363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:30:50.111384 8363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:30:50.111495 8363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:30:50.125296 8220 heartbeater.cc:503] Master 127.6.33.62:41329 was elected leader, sending a full tablet report...
Sep 26 16:30:50 dist-test-slave-khd3 krb5kdc[7968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440250, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.3@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:30:50.141438 8387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:50.113034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:41329 (local address 127.6.33.3:55461)
0926 16:30:50.114083 (+ 1049us) negotiation.cc:107] Waiting for socket to connect
0926 16:30:50.114142 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 16:30:50.117691 (+ 3549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:30:50.118980 (+ 1289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:30:50.118992 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:30:50.119994 (+ 1002us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:30:50.121164 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:50.121189 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:50.122549 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:50.122553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:50.124147 (+ 1594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:50.124162 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:50.125334 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:50.125342 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:50.125615 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:50.126512 (+ 897us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:30:50.126548 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:30:50.130279 (+ 3731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:30:50.135127 (+ 4848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:30:50.135140 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:30:50.135170 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:30:50.135960 (+ 790us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:30:50.136592 (+ 632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:30:50.136597 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:30:50.136601 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:30:50.137003 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:30:50.137702 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:30:50.137710 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:30:50.138187 (+ 477us) client_negotiation.cc:769] Sending connection context
0926 16:30:50.138694 (+ 507us) client_negotiation.cc:240] Negotiation successful
0926 16:30:50.139488 (+ 794us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":739,"thread_start_us":462,"threads_started":1}
I20260926 16:30:50.141717 8108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:50.112507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.3:55461 (local address 127.6.33.62:41329)
0926 16:30:50.112807 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:50.112813 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:50.114283 (+ 1470us) server_negotiation.cc:407] Connection header received
0926 16:30:50.118150 (+ 3867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:50.118157 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:50.118264 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:50.118343 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:30:50.121392 (+ 3049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:50.122365 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:50.124318 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:50.124986 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:50.125140 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:50.130592 (+ 5452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:30:50.130597 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:30:50.130601 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:30:50.130635 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:30:50.134879 (+ 4244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:50.136243 (+ 1364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:50.136249 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:50.136256 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:50.136456 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:50.137127 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:50.137132 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:50.137136 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:50.137580 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:30:50.137800 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:50.141047 (+ 3247us) server_negotiation.cc:299] Negotiation successful
0926 16:30:50.141308 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149}
I20260926 16:30:50.163043 8363 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.3:45795
I20260926 16:30:50.163053 8497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.3:45795 every 8 connection(s)
I20260926 16:30:50.164820 8363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
I20260926 16:30:50.167305 8498 heartbeater.cc:347] Connected to a master server at 127.6.33.62:41329
I20260926 16:30:50.167565 8498 heartbeater.cc:464] Registering TS with master...
I20260926 16:30:50.168423 8498 heartbeater.cc:511] Master 127.6.33.62:41329 requested a full tablet report, sending...
I20260926 16:30:50.169521 8108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:50.151454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.3:33893 (local address 127.6.33.62:41329)
0926 16:30:50.151648 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:50.151655 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:50.152973 (+ 1318us) server_negotiation.cc:407] Connection header received
0926 16:30:50.153197 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:50.153202 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:50.153290 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:50.153408 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:30:50.155189 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:50.156499 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:50.158893 (+ 2394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:50.159583 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:50.159737 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:50.162439 (+ 2702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:30:50.162450 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:30:50.162458 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:30:50.162513 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:30:50.166305 (+ 3792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:50.167114 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:50.167120 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:50.167124 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:50.167308 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:50.167728 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:50.167733 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:50.167736 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:50.168107 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:30:50.168394 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:50.168918 (+ 524us) server_negotiation.cc:299] Negotiation successful
0926 16:30:50.169145 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 16:30:50.169648 8471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:50.151588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:41329 (local address 127.6.33.3:33893)
0926 16:30:50.152847 (+ 1259us) negotiation.cc:107] Waiting for socket to connect
0926 16:30:50.152865 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 16:30:50.153044 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:30:50.153484 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:30:50.153490 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:30:50.154190 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:30:50.154932 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:50.154951 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:50.156718 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:50.156725 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:50.158580 (+ 1855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:50.158601 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:50.159813 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:50.159819 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:50.159991 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:50.160802 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:30:50.160826 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:30:50.162077 (+ 1251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:30:50.166465 (+ 4388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:30:50.166470 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:30:50.166474 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:30:50.166975 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:30:50.167433 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:30:50.167440 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:30:50.167443 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:30:50.167579 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:30:50.168349 (+ 770us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:30:50.168355 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:30:50.168590 (+ 235us) client_negotiation.cc:769] Sending connection context
0926 16:30:50.168754 (+ 164us) client_negotiation.cc:240] Negotiation successful
0926 16:30:50.169144 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1046,"thread_start_us":777,"threads_started":1}
I20260926 16:30:50.170084 8013 ts_manager.cc:194] Registered new tserver with Master: 49a7b8cf178d46f28c018c0bda345304 (127.6.33.3:45795)
I20260926 16:30:50.171200 8013 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.33.3@KRBTEST.COM'} at 127.6.33.3:55461
I20260926 16:30:50.174011 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 8363
I20260926 16:30:50.174191 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/ts-2/wal/instance
I20260926 16:30:50.174638 6276 external_mini_cluster.cc:1428] Setting key a2d70c4b9b7a86b24043220fe7965a79
I20260926 16:30:50.177752 6276 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 16:30:50 dist-test-slave-khd3 krb5kdc[7968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
W20260926 16:30:50.299670 6276 mini_kdc.cc:317] Time spent kinit for test-admin: real 0.120s user 0.002s sys 0.008s
Sep 26 16:30:50 dist-test-slave-khd3 krb5kdc[7968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:30:50.321087 8108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:50.305467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52212 (local address 127.6.33.62:41329)
0926 16:30:50.305726 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:50.305734 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:50.305929 (+ 195us) server_negotiation.cc:407] Connection header received
0926 16:30:50.306165 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:50.306172 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:50.306259 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:50.306379 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:30:50.309059 (+ 2680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:50.309806 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:50.311125 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:50.311545 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:50.311695 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:50.315800 (+ 4105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:30:50.315806 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:30:50.315809 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:30:50.315847 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:30:50.318551 (+ 2704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:50.319237 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:50.319243 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:50.319247 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:50.319401 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:50.319764 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:50.319767 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:50.319769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:50.320029 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:30:50.320205 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:50.320545 (+ 340us) server_negotiation.cc:299] Negotiation successful
0926 16:30:50.320709 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260926 16:30:50.325408 8013 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52212:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:30:50.332405 8013 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:30:50.365985 8516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:50.347025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50020 (local address 127.6.33.1:38069)
0926 16:30:50.357756 (+ 10731us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:50.357779 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:50.358292 (+ 513us) server_negotiation.cc:407] Connection header received
0926 16:30:50.358536 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:50.358542 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:50.358782 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:50.359004 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:30:50.359693 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:50.360704 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:50.362580 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:50.364697 (+ 2117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:50.364871 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:50.365085 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:50.365488 (+ 403us) server_negotiation.cc:299] Negotiation successful
0926 16:30:50.365649 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10581,"thread_start_us":10450,"threads_started":1}
I20260926 16:30:50.366094 8511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:50.346118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:38069 (local address 127.0.0.1:50020)
0926 16:30:50.358153 (+ 12035us) negotiation.cc:107] Waiting for socket to connect
0926 16:30:50.358170 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 16:30:50.358342 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:30:50.359030 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:30:50.359037 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:30:50.359070 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:30:50.359522 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:50.359533 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:50.360882 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:50.360889 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:50.362426 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:50.362438 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:50.364947 (+ 2509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:50.364958 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:50.365191 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:50.365283 (+ 92us) client_negotiation.cc:769] Sending connection context
0926 16:30:50.365435 (+ 152us) client_negotiation.cc:240] Negotiation successful
0926 16:30:50.365650 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11905,"mutex_wait_us":3,"thread_start_us":11730,"threads_started":1}
I20260926 16:30:50.370963 8513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:50.346233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51128 (local address 127.6.33.2:37269)
0926 16:30:50.363020 (+ 16787us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:50.363029 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:50.363060 (+ 31us) server_negotiation.cc:407] Connection header received
0926 16:30:50.363158 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:50.363166 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:50.363457 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:50.363665 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:30:50.364454 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:50.365795 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:50.368578 (+ 2783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:50.369967 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:50.370109 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:50.370357 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:50.370525 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 16:30:50.370680 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16615,"thread_start_us":16495,"threads_started":1}
I20260926 16:30:50.371048 8514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:50.346105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.2:37269 (local address 127.0.0.1:51128)
0926 16:30:50.356183 (+ 10078us) negotiation.cc:107] Waiting for socket to connect
0926 16:30:50.356231 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 16:30:50.356477 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:30:50.363744 (+ 7267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:30:50.363748 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:30:50.363772 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:30:50.364221 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:50.364237 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:50.365960 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:50.365966 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:50.368401 (+ 2435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:50.368415 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:50.370158 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:50.370164 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:50.370331 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:50.370365 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 16:30:50.370486 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 16:30:50.370631 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9759,"mutex_wait_us":39,"thread_start_us":17081,"threads_started":1}
I20260926 16:30:50.370558 8150 tablet_service.cc:1467] Processing CreateTablet for tablet 1dc7e5c0faee4950ad0e763903a8084e (DEFAULT_TABLE table=test-table [id=e2767078a9d340e4943f5cc73ad92036]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:30:50.372495 8150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dc7e5c0faee4950ad0e763903a8084e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:30:50.372678 8512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:50.346502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.3:45795 (local address 127.0.0.1:34310)
0926 16:30:50.363442 (+ 16940us) negotiation.cc:107] Waiting for socket to connect
0926 16:30:50.363462 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 16:30:50.363588 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:30:50.364493 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:30:50.364497 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:30:50.364523 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:30:50.364900 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:50.364912 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:50.366973 (+ 2061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:50.366978 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:50.369153 (+ 2175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:50.369176 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:50.371869 (+ 2693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:50.371875 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:50.372047 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:50.372080 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 16:30:50.372160 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 16:30:50.372297 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16819,"mutex_wait_us":1,"thread_start_us":9193,"threads_started":1}
I20260926 16:30:50.372721 8515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:50.346929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34310 (local address 127.6.33.3:45795)
0926 16:30:50.357330 (+ 10401us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:50.357343 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:50.363688 (+ 6345us) server_negotiation.cc:407] Connection header received
0926 16:30:50.363794 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:50.363803 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:50.364065 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:50.364275 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:30:50.365131 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:50.366814 (+ 1683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:50.369334 (+ 2520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:50.371674 (+ 2340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:50.371842 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:50.372067 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:50.372222 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 16:30:50.372380 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10162,"thread_start_us":9968,"threads_started":1}
I20260926 16:30:50.375006 8289 tablet_service.cc:1467] Processing CreateTablet for tablet 1dc7e5c0faee4950ad0e763903a8084e (DEFAULT_TABLE table=test-table [id=e2767078a9d340e4943f5cc73ad92036]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:30:50.375815 8428 tablet_service.cc:1467] Processing CreateTablet for tablet 1dc7e5c0faee4950ad0e763903a8084e (DEFAULT_TABLE table=test-table [id=e2767078a9d340e4943f5cc73ad92036]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:30:50.376406 8289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dc7e5c0faee4950ad0e763903a8084e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:30:50.377068 8428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dc7e5c0faee4950ad0e763903a8084e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:30:50.389142 8517 tablet_bootstrap.cc:492] T 1dc7e5c0faee4950ad0e763903a8084e P 9cf28505064b4b4ba7581338c203ee66: Bootstrap starting.
I20260926 16:30:50.390533 8518 tablet_bootstrap.cc:492] T 1dc7e5c0faee4950ad0e763903a8084e P 49a7b8cf178d46f28c018c0bda345304: Bootstrap starting.
I20260926 16:30:50.393421 8517 tablet_bootstrap.cc:654] T 1dc7e5c0faee4950ad0e763903a8084e P 9cf28505064b4b4ba7581338c203ee66: Neither blocks nor log segments found. Creating new log.
I20260926 16:30:50.394187 8518 tablet_bootstrap.cc:654] T 1dc7e5c0faee4950ad0e763903a8084e P 49a7b8cf178d46f28c018c0bda345304: Neither blocks nor log segments found. Creating new log.
I20260926 16:30:50.394352 8519 tablet_bootstrap.cc:492] T 1dc7e5c0faee4950ad0e763903a8084e P 6a4cdd1c228144b1a0a7bd99cab216cf: Bootstrap starting.
I20260926 16:30:50.394845 8517 log.cc:824] T 1dc7e5c0faee4950ad0e763903a8084e P 9cf28505064b4b4ba7581338c203ee66: Log is configured to *not* fsync() on all Append() calls
I20260926 16:30:50.395437 8518 log.cc:824] T 1dc7e5c0faee4950ad0e763903a8084e P 49a7b8cf178d46f28c018c0bda345304: Log is configured to *not* fsync() on all Append() calls
I20260926 16:30:50.398325 8517 tablet_bootstrap.cc:492] T 1dc7e5c0faee4950ad0e763903a8084e P 9cf28505064b4b4ba7581338c203ee66: No bootstrap required, opened a new log
I20260926 16:30:50.398495 8519 tablet_bootstrap.cc:654] T 1dc7e5c0faee4950ad0e763903a8084e P 6a4cdd1c228144b1a0a7bd99cab216cf: Neither blocks nor log segments found. Creating new log.
I20260926 16:30:50.398656 8517 ts_tablet_manager.cc:1397] T 1dc7e5c0faee4950ad0e763903a8084e P 9cf28505064b4b4ba7581338c203ee66: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260926 16:30:50.399931 8519 log.cc:824] T 1dc7e5c0faee4950ad0e763903a8084e P 6a4cdd1c228144b1a0a7bd99cab216cf: Log is configured to *not* fsync() on all Append() calls
I20260926 16:30:50.403764 8518 tablet_bootstrap.cc:492] T 1dc7e5c0faee4950ad0e763903a8084e P 49a7b8cf178d46f28c018c0bda345304: No bootstrap required, opened a new log
I20260926 16:30:50.404098 8518 ts_tablet_manager.cc:1397] T 1dc7e5c0faee4950ad0e763903a8084e P 49a7b8cf178d46f28c018c0bda345304: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.000s
I20260926 16:30:50.404134 8519 tablet_bootstrap.cc:492] T 1dc7e5c0faee4950ad0e763903a8084e P 6a4cdd1c228144b1a0a7bd99cab216cf: No bootstrap required, opened a new log
I20260926 16:30:50.404377 8519 ts_tablet_manager.cc:1397] T 1dc7e5c0faee4950ad0e763903a8084e P 6a4cdd1c228144b1a0a7bd99cab216cf: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260926 16:30:50.409973 8517 raft_consensus.cc:348] T 1dc7e5c0faee4950ad0e763903a8084e P 9cf28505064b4b4ba7581338c203ee66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37269 } } peers { permanent_uuid: "9cf28505064b4b4ba7581338c203ee66" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38069 } } peers { permanent_uuid: "49a7b8cf178d46f28c018c0bda345304" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45795 } }
I20260926 16:30:50.410468 8517 raft_consensus.cc:374] T 1dc7e5c0faee4950ad0e763903a8084e P 9cf28505064b4b4ba7581338c203ee66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:30:50.410573 8517 raft_consensus.cc:729] T 1dc7e5c0faee4950ad0e763903a8084e P 9cf28505064b4b4ba7581338c203ee66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cf28505064b4b4ba7581338c203ee66, State: Initialized, Role: FOLLOWER
I20260926 16:30:50.411255 8517 consensus_queue.cc:260] T 1dc7e5c0faee4950ad0e763903a8084e P 9cf28505064b4b4ba7581338c203ee66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37269 } } peers { permanent_uuid: "9cf28505064b4b4ba7581338c203ee66" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38069 } } peers { permanent_uuid: "49a7b8cf178d46f28c018c0bda345304" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45795 } }
I20260926 16:30:50.412405 8518 raft_consensus.cc:348] T 1dc7e5c0faee4950ad0e763903a8084e P 49a7b8cf178d46f28c018c0bda345304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37269 } } peers { permanent_uuid: "9cf28505064b4b4ba7581338c203ee66" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38069 } } peers { permanent_uuid: "49a7b8cf178d46f28c018c0bda345304" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45795 } }
I20260926 16:30:50.412853 8518 raft_consensus.cc:374] T 1dc7e5c0faee4950ad0e763903a8084e P 49a7b8cf178d46f28c018c0bda345304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:30:50.412951 8518 raft_consensus.cc:729] T 1dc7e5c0faee4950ad0e763903a8084e P 49a7b8cf178d46f28c018c0bda345304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49a7b8cf178d46f28c018c0bda345304, State: Initialized, Role: FOLLOWER
I20260926 16:30:50.413141 8517 ts_tablet_manager.cc:1428] T 1dc7e5c0faee4950ad0e763903a8084e P 9cf28505064b4b4ba7581338c203ee66: Time spent starting tablet: real 0.014s user 0.016s sys 0.000s
I20260926 16:30:50.413597 8518 consensus_queue.cc:260] T 1dc7e5c0faee4950ad0e763903a8084e P 49a7b8cf178d46f28c018c0bda345304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37269 } } peers { permanent_uuid: "9cf28505064b4b4ba7581338c203ee66" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38069 } } peers { permanent_uuid: "49a7b8cf178d46f28c018c0bda345304" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45795 } }
I20260926 16:30:50.414727 8519 raft_consensus.cc:348] T 1dc7e5c0faee4950ad0e763903a8084e P 6a4cdd1c228144b1a0a7bd99cab216cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37269 } } peers { permanent_uuid: "9cf28505064b4b4ba7581338c203ee66" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38069 } } peers { permanent_uuid: "49a7b8cf178d46f28c018c0bda345304" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45795 } }
I20260926 16:30:50.415128 8519 raft_consensus.cc:374] T 1dc7e5c0faee4950ad0e763903a8084e P 6a4cdd1c228144b1a0a7bd99cab216cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:30:50.415230 8518 ts_tablet_manager.cc:1428] T 1dc7e5c0faee4950ad0e763903a8084e P 49a7b8cf178d46f28c018c0bda345304: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260926 16:30:50.415226 8519 raft_consensus.cc:729] T 1dc7e5c0faee4950ad0e763903a8084e P 6a4cdd1c228144b1a0a7bd99cab216cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a4cdd1c228144b1a0a7bd99cab216cf, State: Initialized, Role: FOLLOWER
I20260926 16:30:50.415851 8519 consensus_queue.cc:260] T 1dc7e5c0faee4950ad0e763903a8084e P 6a4cdd1c228144b1a0a7bd99cab216cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37269 } } peers { permanent_uuid: "9cf28505064b4b4ba7581338c203ee66" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38069 } } peers { permanent_uuid: "49a7b8cf178d46f28c018c0bda345304" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45795 } }
I20260926 16:30:50.416671 8498 heartbeater.cc:503] Master 127.6.33.62:41329 was elected leader, sending a full tablet report...
W20260926 16:30:50.416983 8499 tablet.cc:2369] T 1dc7e5c0faee4950ad0e763903a8084e P 49a7b8cf178d46f28c018c0bda345304: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:30:50.417972 8359 heartbeater.cc:503] Master 127.6.33.62:41329 was elected leader, sending a full tablet report...
I20260926 16:30:50.418489 8519 ts_tablet_manager.cc:1428] T 1dc7e5c0faee4950ad0e763903a8084e P 6a4cdd1c228144b1a0a7bd99cab216cf: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260926 16:30:50.549541 8525 raft_consensus.cc:482] T 1dc7e5c0faee4950ad0e763903a8084e P 6a4cdd1c228144b1a0a7bd99cab216cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:30:50.549909 8525 raft_consensus.cc:504] T 1dc7e5c0faee4950ad0e763903a8084e P 6a4cdd1c228144b1a0a7bd99cab216cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37269 } } peers { permanent_uuid: "9cf28505064b4b4ba7581338c203ee66" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38069 } } peers { permanent_uuid: "49a7b8cf178d46f28c018c0bda345304" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45795 } }
I20260926 16:30:50.551707 8525 leader_election.cc:288] T 1dc7e5c0faee4950ad0e763903a8084e P 6a4cdd1c228144b1a0a7bd99cab216cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9cf28505064b4b4ba7581338c203ee66 (127.6.33.1:38069), 49a7b8cf178d46f28c018c0bda345304 (127.6.33.3:45795)
I20260926 16:30:50.557047 8523 raft_consensus.cc:482] T 1dc7e5c0faee4950ad0e763903a8084e P 9cf28505064b4b4ba7581338c203ee66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:30:50.557456 8523 raft_consensus.cc:504] T 1dc7e5c0faee4950ad0e763903a8084e P 9cf28505064b4b4ba7581338c203ee66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37269 } } peers { permanent_uuid: "9cf28505064b4b4ba7581338c203ee66" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38069 } } peers { permanent_uuid: "49a7b8cf178d46f28c018c0bda345304" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45795 } }
I20260926 16:30:50.559330 8515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:50.552184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:47529 (local address 127.6.33.3:45795)
0926 16:30:50.552417 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:50.552427 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:50.553210 (+ 783us) server_negotiation.cc:407] Connection header received
0926 16:30:50.553459 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:50.553462 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:50.553522 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:50.553622 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:30:50.554230 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:50.554948 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:50.557162 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:50.558220 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:50.558381 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:50.558516 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:50.558928 (+ 412us) server_negotiation.cc:299] Negotiation successful
0926 16:30:50.559069 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 16:30:50.559571 8516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:50.552120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:45725 (local address 127.6.33.1:38069)
0926 16:30:50.552417 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:50.552426 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:50.553202 (+ 776us) server_negotiation.cc:407] Connection header received
0926 16:30:50.553460 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:50.553464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:50.553535 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:50.553637 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:30:50.554324 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:50.555000 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:50.557213 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:50.558534 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:50.558712 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:50.558833 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:50.559166 (+ 333us) server_negotiation.cc:299] Negotiation successful
0926 16:30:50.559327 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260926 16:30:50.559847 8526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:50.552121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.3:45795 (local address 127.6.33.2:47529)
0926 16:30:50.553088 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0926 16:30:50.553103 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 16:30:50.553304 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:30:50.553638 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:30:50.553642 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:30:50.553665 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:30:50.554060 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:50.554073 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:50.555083 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:50.555091 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:50.557000 (+ 1909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:50.557014 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:50.558453 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:50.558459 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:50.558689 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:50.558745 (+ 56us) client_negotiation.cc:769] Sending connection context
0926 16:30:50.558890 (+ 145us) client_negotiation.cc:240] Negotiation successful
0926 16:30:50.559045 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":855,"mutex_wait_us":59,"spinlock_wait_cycles":52480,"thread_start_us":650,"threads_started":1}
I20260926 16:30:50.560436 8527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:50.552086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:38069 (local address 127.6.33.2:45725)
0926 16:30:50.553067 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0926 16:30:50.553087 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 16:30:50.553306 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:30:50.553653 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:30:50.553657 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:30:50.553680 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:30:50.554176 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:50.554184 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:50.555141 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:50.555151 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:50.557008 (+ 1857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:50.557027 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:50.558766 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:50.558771 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:50.558950 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:50.558979 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 16:30:50.559886 (+ 907us) client_negotiation.cc:240] Negotiation successful
0926 16:30:50.560042 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":824,"mutex_wait_us":53,"spinlock_wait_cycles":17792,"thread_start_us":545,"threads_started":1}
I20260926 16:30:50.560714 8523 leader_election.cc:288] T 1dc7e5c0faee4950ad0e763903a8084e P 9cf28505064b4b4ba7581338c203ee66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a4cdd1c228144b1a0a7bd99cab216cf (127.6.33.2:37269), 49a7b8cf178d46f28c018c0bda345304 (127.6.33.3:45795)
I20260926 16:30:50.560985 8448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dc7e5c0faee4950ad0e763903a8084e" candidate_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49a7b8cf178d46f28c018c0bda345304" is_pre_election: true
I20260926 16:30:50.561626 8448 raft_consensus.cc:2454] T 1dc7e5c0faee4950ad0e763903a8084e P 49a7b8cf178d46f28c018c0bda345304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a4cdd1c228144b1a0a7bd99cab216cf in term 0.
I20260926 16:30:50.562424 8240 leader_election.cc:302] T 1dc7e5c0faee4950ad0e763903a8084e P 6a4cdd1c228144b1a0a7bd99cab216cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49a7b8cf178d46f28c018c0bda345304, 6a4cdd1c228144b1a0a7bd99cab216cf; no voters:
I20260926 16:30:50.562887 8525 raft_consensus.cc:2781] T 1dc7e5c0faee4950ad0e763903a8084e P 6a4cdd1c228144b1a0a7bd99cab216cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:30:50.563045 8525 raft_consensus.cc:482] T 1dc7e5c0faee4950ad0e763903a8084e P 6a4cdd1c228144b1a0a7bd99cab216cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:30:50.563144 8525 raft_consensus.cc:3037] T 1dc7e5c0faee4950ad0e763903a8084e P 6a4cdd1c228144b1a0a7bd99cab216cf [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:30:50.568485 8513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:50.561683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:42475 (local address 127.6.33.2:37269)
0926 16:30:50.561866 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:50.561872 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:50.561961 (+ 89us) server_negotiation.cc:407] Connection header received
0926 16:30:50.562268 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:50.562274 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:50.562345 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:50.562443 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:30:50.563457 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:50.564357 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:50.566684 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:50.567501 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:50.567660 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:50.567787 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:50.568073 (+ 286us) server_negotiation.cc:299] Negotiation successful
0926 16:30:50.568198 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":68}
I20260926 16:30:50.568536 8528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:50.561219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.2:37269 (local address 127.6.33.1:42475)
0926 16:30:50.561862 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0926 16:30:50.561882 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 16:30:50.562080 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:30:50.562502 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:30:50.562510 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:30:50.562551 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:30:50.563274 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:50.563293 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:50.564498 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:50.564505 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:50.566506 (+ 2001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:50.566521 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:50.567745 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:50.567751 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:50.567912 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:50.567937 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 16:30:50.568047 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 16:30:50.568191 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":966,"threads_started":1}
I20260926 16:30:50.569737 8309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dc7e5c0faee4950ad0e763903a8084e" candidate_uuid: "9cf28505064b4b4ba7581338c203ee66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" is_pre_election: true
I20260926 16:30:50.572962 8170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dc7e5c0faee4950ad0e763903a8084e" candidate_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cf28505064b4b4ba7581338c203ee66" is_pre_election: true
I20260926 16:30:50.573443 8170 raft_consensus.cc:2454] T 1dc7e5c0faee4950ad0e763903a8084e P 9cf28505064b4b4ba7581338c203ee66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a4cdd1c228144b1a0a7bd99cab216cf in term 0.
I20260926 16:30:50.574198 8528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:50.561824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.3:45795 (local address 127.6.33.1:48411)
0926 16:30:50.568962 (+ 7138us) negotiation.cc:107] Waiting for socket to connect
0926 16:30:50.568985 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 16:30:50.569165 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:30:50.569571 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:30:50.569576 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:30:50.569602 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:30:50.569777 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:50.569784 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:50.570878 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:50.570885 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:50.572154 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:50.572166 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:50.573376 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:50.573381 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:50.573606 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:50.573639 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 16:30:50.573757 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 16:30:50.573935 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7005,"thread_start_us":10249,"threads_started":1}
I20260926 16:30:50.574203 8515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:50.562970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:48411 (local address 127.6.33.3:45795)
0926 16:30:50.563185 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:50.563194 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:50.569058 (+ 5864us) server_negotiation.cc:407] Connection header received
0926 16:30:50.569374 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:50.569380 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:50.569450 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:50.569545 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:30:50.569910 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:50.570750 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:50.572329 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:50.573172 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:50.573333 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:50.573418 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:50.573853 (+ 435us) server_negotiation.cc:299] Negotiation successful
0926 16:30:50.574002 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 16:30:50.574932 8448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dc7e5c0faee4950ad0e763903a8084e" candidate_uuid: "9cf28505064b4b4ba7581338c203ee66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49a7b8cf178d46f28c018c0bda345304" is_pre_election: true
I20260926 16:30:50.575142 8448 raft_consensus.cc:2454] T 1dc7e5c0faee4950ad0e763903a8084e P 49a7b8cf178d46f28c018c0bda345304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cf28505064b4b4ba7581338c203ee66 in term 0.
I20260926 16:30:50.575680 8100 leader_election.cc:302] T 1dc7e5c0faee4950ad0e763903a8084e P 9cf28505064b4b4ba7581338c203ee66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49a7b8cf178d46f28c018c0bda345304, 9cf28505064b4b4ba7581338c203ee66; no voters:
I20260926 16:30:50.576030 8523 raft_consensus.cc:2781] T 1dc7e5c0faee4950ad0e763903a8084e P 9cf28505064b4b4ba7581338c203ee66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:30:50.576186 8523 raft_consensus.cc:482] T 1dc7e5c0faee4950ad0e763903a8084e P 9cf28505064b4b4ba7581338c203ee66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:30:50.576321 8523 raft_consensus.cc:3037] T 1dc7e5c0faee4950ad0e763903a8084e P 9cf28505064b4b4ba7581338c203ee66 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:30:50.598861 8523 raft_consensus.cc:504] T 1dc7e5c0faee4950ad0e763903a8084e P 9cf28505064b4b4ba7581338c203ee66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37269 } } peers { permanent_uuid: "9cf28505064b4b4ba7581338c203ee66" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38069 } } peers { permanent_uuid: "49a7b8cf178d46f28c018c0bda345304" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45795 } }
I20260926 16:30:50.598861 8525 raft_consensus.cc:504] T 1dc7e5c0faee4950ad0e763903a8084e P 6a4cdd1c228144b1a0a7bd99cab216cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37269 } } peers { permanent_uuid: "9cf28505064b4b4ba7581338c203ee66" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38069 } } peers { permanent_uuid: "49a7b8cf178d46f28c018c0bda345304" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45795 } }
I20260926 16:30:50.599428 8309 raft_consensus.cc:2379] T 1dc7e5c0faee4950ad0e763903a8084e P 6a4cdd1c228144b1a0a7bd99cab216cf [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9cf28505064b4b4ba7581338c203ee66 in current term 1: Already voted for candidate 6a4cdd1c228144b1a0a7bd99cab216cf in this term.
I20260926 16:30:50.599853 8523 leader_election.cc:288] T 1dc7e5c0faee4950ad0e763903a8084e P 9cf28505064b4b4ba7581338c203ee66 [CANDIDATE]: Term 1 election: Requested vote from peers 6a4cdd1c228144b1a0a7bd99cab216cf (127.6.33.2:37269), 49a7b8cf178d46f28c018c0bda345304 (127.6.33.3:45795)
I20260926 16:30:50.600672 8525 leader_election.cc:288] T 1dc7e5c0faee4950ad0e763903a8084e P 6a4cdd1c228144b1a0a7bd99cab216cf [CANDIDATE]: Term 1 election: Requested vote from peers 9cf28505064b4b4ba7581338c203ee66 (127.6.33.1:38069), 49a7b8cf178d46f28c018c0bda345304 (127.6.33.3:45795)
I20260926 16:30:50.601239 8448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dc7e5c0faee4950ad0e763903a8084e" candidate_uuid: "9cf28505064b4b4ba7581338c203ee66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49a7b8cf178d46f28c018c0bda345304"
I20260926 16:30:50.601495 8448 raft_consensus.cc:3037] T 1dc7e5c0faee4950ad0e763903a8084e P 49a7b8cf178d46f28c018c0bda345304 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:30:50.602003 8309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dc7e5c0faee4950ad0e763903a8084e" candidate_uuid: "9cf28505064b4b4ba7581338c203ee66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf"
I20260926 16:30:50.602013 8447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dc7e5c0faee4950ad0e763903a8084e" candidate_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49a7b8cf178d46f28c018c0bda345304"
I20260926 16:30:50.602246 8309 raft_consensus.cc:2379] T 1dc7e5c0faee4950ad0e763903a8084e P 6a4cdd1c228144b1a0a7bd99cab216cf [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9cf28505064b4b4ba7581338c203ee66 in current term 1: Already voted for candidate 6a4cdd1c228144b1a0a7bd99cab216cf in this term.
I20260926 16:30:50.603051 8170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dc7e5c0faee4950ad0e763903a8084e" candidate_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cf28505064b4b4ba7581338c203ee66"
I20260926 16:30:50.603349 8170 raft_consensus.cc:2379] T 1dc7e5c0faee4950ad0e763903a8084e P 9cf28505064b4b4ba7581338c203ee66 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6a4cdd1c228144b1a0a7bd99cab216cf in current term 1: Already voted for candidate 9cf28505064b4b4ba7581338c203ee66 in this term.
I20260926 16:30:50.603946 8242 leader_election.cc:302] T 1dc7e5c0faee4950ad0e763903a8084e P 6a4cdd1c228144b1a0a7bd99cab216cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a4cdd1c228144b1a0a7bd99cab216cf; no voters: 49a7b8cf178d46f28c018c0bda345304, 9cf28505064b4b4ba7581338c203ee66
I20260926 16:30:50.604302 8525 raft_consensus.cc:2726] T 1dc7e5c0faee4950ad0e763903a8084e P 6a4cdd1c228144b1a0a7bd99cab216cf [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20260926 16:30:50.616726 8221 tablet.cc:2369] T 1dc7e5c0faee4950ad0e763903a8084e P 9cf28505064b4b4ba7581338c203ee66: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 16:30:50.628549 8360 tablet.cc:2369] T 1dc7e5c0faee4950ad0e763903a8084e P 6a4cdd1c228144b1a0a7bd99cab216cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:30:50.707823 8448 raft_consensus.cc:2454] T 1dc7e5c0faee4950ad0e763903a8084e P 49a7b8cf178d46f28c018c0bda345304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cf28505064b4b4ba7581338c203ee66 in term 1.
I20260926 16:30:50.708484 8100 leader_election.cc:302] T 1dc7e5c0faee4950ad0e763903a8084e P 9cf28505064b4b4ba7581338c203ee66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49a7b8cf178d46f28c018c0bda345304, 9cf28505064b4b4ba7581338c203ee66; no voters: 6a4cdd1c228144b1a0a7bd99cab216cf
I20260926 16:30:50.708992 8523 raft_consensus.cc:2781] T 1dc7e5c0faee4950ad0e763903a8084e P 9cf28505064b4b4ba7581338c203ee66 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:30:50.709559 8523 raft_consensus.cc:686] T 1dc7e5c0faee4950ad0e763903a8084e P 9cf28505064b4b4ba7581338c203ee66 [term 1 LEADER]: Becoming Leader. State: Replica: 9cf28505064b4b4ba7581338c203ee66, State: Running, Role: LEADER
I20260926 16:30:50.710172 8523 consensus_queue.cc:237] T 1dc7e5c0faee4950ad0e763903a8084e P 9cf28505064b4b4ba7581338c203ee66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37269 } } peers { permanent_uuid: "9cf28505064b4b4ba7581338c203ee66" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38069 } } peers { permanent_uuid: "49a7b8cf178d46f28c018c0bda345304" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45795 } }
I20260926 16:30:50.715655 8011 catalog_manager.cc:5495] T 1dc7e5c0faee4950ad0e763903a8084e P 9cf28505064b4b4ba7581338c203ee66 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9cf28505064b4b4ba7581338c203ee66 (127.6.33.1). New cstate: current_term: 1 leader_uuid: "9cf28505064b4b4ba7581338c203ee66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cf28505064b4b4ba7581338c203ee66" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38069 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49a7b8cf178d46f28c018c0bda345304" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45795 } health_report { overall_health: UNKNOWN } } }
Sep 26 16:30:50 dist-test-slave-khd3 krb5kdc[7968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440248, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.62@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:30:50.781684 8108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:50.767345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52218 (local address 127.6.33.62:41329)
0926 16:30:50.767577 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:50.767586 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:50.768196 (+ 610us) server_negotiation.cc:407] Connection header received
0926 16:30:50.768339 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:50.768343 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:50.768398 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:50.768481 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:30:50.769775 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:50.770511 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:50.771747 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:50.772162 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:50.772278 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:50.775561 (+ 3283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:30:50.775565 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:30:50.775569 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:30:50.775595 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:30:50.778914 (+ 3319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:50.779635 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:50.779643 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:50.779648 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:50.779778 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:50.780136 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:50.780144 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:50.780147 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:50.780422 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:30:50.780570 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:50.781026 (+ 456us) server_negotiation.cc:299] Negotiation successful
0926 16:30:50.781368 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":145,"server-negotiator.queue_time_us":78}
I20260926 16:30:50.781718 8537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:50.767418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:41329 (local address 127.0.0.1:52218)
0926 16:30:50.768060 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0926 16:30:50.768076 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:30:50.768234 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:30:50.768575 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:30:50.768578 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:30:50.769377 (+ 799us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:30:50.769625 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:50.769634 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:50.770632 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:50.770635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:50.771606 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:50.771617 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:50.772367 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:50.772372 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:50.772499 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:50.773156 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:30:50.773179 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:30:50.775378 (+ 2199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:30:50.779060 (+ 3682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:30:50.779077 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:30:50.779085 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:30:50.779508 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:30:50.779885 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:30:50.779891 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:30:50.779894 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:30:50.780036 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:30:50.780548 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:30:50.780556 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:30:50.780723 (+ 167us) client_negotiation.cc:769] Sending connection context
0926 16:30:50.780940 (+ 217us) client_negotiation.cc:240] Negotiation successful
0926 16:30:50.781248 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":384,"threads_started":1}
I20260926 16:30:50.787526 8011 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.6.33.62@KRBTEST.COM'} at 127.0.0.1:52218:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 16:30:50.788865 8011 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:30:50.795799 8150 tablet_service.cc:1467] Processing CreateTablet for tablet 32a51e7e22684860823bb976e508bd71 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=11ae351d15df4831b9200d58021b5a66]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 16:30:50.795799 8289 tablet_service.cc:1467] Processing CreateTablet for tablet 32a51e7e22684860823bb976e508bd71 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=11ae351d15df4831b9200d58021b5a66]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 16:30:50.796838 8289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32a51e7e22684860823bb976e508bd71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:30:50.796859 8150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32a51e7e22684860823bb976e508bd71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:30:50.796325 8428 tablet_service.cc:1467] Processing CreateTablet for tablet 32a51e7e22684860823bb976e508bd71 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=11ae351d15df4831b9200d58021b5a66]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 16:30:50.797210 8428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32a51e7e22684860823bb976e508bd71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:30:50.802709 8518 tablet_bootstrap.cc:492] T 32a51e7e22684860823bb976e508bd71 P 49a7b8cf178d46f28c018c0bda345304: Bootstrap starting.
I20260926 16:30:50.804064 8519 tablet_bootstrap.cc:492] T 32a51e7e22684860823bb976e508bd71 P 6a4cdd1c228144b1a0a7bd99cab216cf: Bootstrap starting.
I20260926 16:30:50.804291 8517 tablet_bootstrap.cc:492] T 32a51e7e22684860823bb976e508bd71 P 9cf28505064b4b4ba7581338c203ee66: Bootstrap starting.
I20260926 16:30:50.804461 8518 tablet_bootstrap.cc:654] T 32a51e7e22684860823bb976e508bd71 P 49a7b8cf178d46f28c018c0bda345304: Neither blocks nor log segments found. Creating new log.
I20260926 16:30:50.805805 8517 tablet_bootstrap.cc:654] T 32a51e7e22684860823bb976e508bd71 P 9cf28505064b4b4ba7581338c203ee66: Neither blocks nor log segments found. Creating new log.
I20260926 16:30:50.805871 8519 tablet_bootstrap.cc:654] T 32a51e7e22684860823bb976e508bd71 P 6a4cdd1c228144b1a0a7bd99cab216cf: Neither blocks nor log segments found. Creating new log.
I20260926 16:30:50.807134 8518 tablet_bootstrap.cc:492] T 32a51e7e22684860823bb976e508bd71 P 49a7b8cf178d46f28c018c0bda345304: No bootstrap required, opened a new log
I20260926 16:30:50.807394 8518 ts_tablet_manager.cc:1397] T 32a51e7e22684860823bb976e508bd71 P 49a7b8cf178d46f28c018c0bda345304: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 16:30:50.808640 8519 tablet_bootstrap.cc:492] T 32a51e7e22684860823bb976e508bd71 P 6a4cdd1c228144b1a0a7bd99cab216cf: No bootstrap required, opened a new log
I20260926 16:30:50.808477 8518 raft_consensus.cc:348] T 32a51e7e22684860823bb976e508bd71 P 49a7b8cf178d46f28c018c0bda345304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf28505064b4b4ba7581338c203ee66" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38069 } } peers { permanent_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37269 } } peers { permanent_uuid: "49a7b8cf178d46f28c018c0bda345304" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45795 } }
I20260926 16:30:50.808842 8519 ts_tablet_manager.cc:1397] T 32a51e7e22684860823bb976e508bd71 P 6a4cdd1c228144b1a0a7bd99cab216cf: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 16:30:50.808857 8518 raft_consensus.cc:374] T 32a51e7e22684860823bb976e508bd71 P 49a7b8cf178d46f28c018c0bda345304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:30:50.808960 8518 raft_consensus.cc:729] T 32a51e7e22684860823bb976e508bd71 P 49a7b8cf178d46f28c018c0bda345304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49a7b8cf178d46f28c018c0bda345304, State: Initialized, Role: FOLLOWER
I20260926 16:30:50.809273 8518 consensus_queue.cc:260] T 32a51e7e22684860823bb976e508bd71 P 49a7b8cf178d46f28c018c0bda345304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf28505064b4b4ba7581338c203ee66" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38069 } } peers { permanent_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37269 } } peers { permanent_uuid: "49a7b8cf178d46f28c018c0bda345304" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45795 } }
I20260926 16:30:50.810056 8518 ts_tablet_manager.cc:1428] T 32a51e7e22684860823bb976e508bd71 P 49a7b8cf178d46f28c018c0bda345304: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260926 16:30:50.809870 8524 tablet_replica.cc:439] T 32a51e7e22684860823bb976e508bd71 P 49a7b8cf178d46f28c018c0bda345304: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf28505064b4b4ba7581338c203ee66" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38069 } } peers { permanent_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37269 } } peers { permanent_uuid: "49a7b8cf178d46f28c018c0bda345304" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45795 } } }
I20260926 16:30:50.811123 8524 tablet_replica.cc:442] T 32a51e7e22684860823bb976e508bd71 P 49a7b8cf178d46f28c018c0bda345304: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:30:50.811789 8517 tablet_bootstrap.cc:492] T 32a51e7e22684860823bb976e508bd71 P 9cf28505064b4b4ba7581338c203ee66: No bootstrap required, opened a new log
I20260926 16:30:50.811950 8517 ts_tablet_manager.cc:1397] T 32a51e7e22684860823bb976e508bd71 P 9cf28505064b4b4ba7581338c203ee66: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260926 16:30:50.812865 8517 raft_consensus.cc:348] T 32a51e7e22684860823bb976e508bd71 P 9cf28505064b4b4ba7581338c203ee66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf28505064b4b4ba7581338c203ee66" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38069 } } peers { permanent_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37269 } } peers { permanent_uuid: "49a7b8cf178d46f28c018c0bda345304" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45795 } }
I20260926 16:30:50.813138 8517 raft_consensus.cc:374] T 32a51e7e22684860823bb976e508bd71 P 9cf28505064b4b4ba7581338c203ee66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:30:50.813231 8517 raft_consensus.cc:729] T 32a51e7e22684860823bb976e508bd71 P 9cf28505064b4b4ba7581338c203ee66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cf28505064b4b4ba7581338c203ee66, State: Initialized, Role: FOLLOWER
I20260926 16:30:50.813469 8517 consensus_queue.cc:260] T 32a51e7e22684860823bb976e508bd71 P 9cf28505064b4b4ba7581338c203ee66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf28505064b4b4ba7581338c203ee66" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38069 } } peers { permanent_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37269 } } peers { permanent_uuid: "49a7b8cf178d46f28c018c0bda345304" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45795 } }
I20260926 16:30:50.814069 8517 ts_tablet_manager.cc:1428] T 32a51e7e22684860823bb976e508bd71 P 9cf28505064b4b4ba7581338c203ee66: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 16:30:50.813942 8523 tablet_replica.cc:439] T 32a51e7e22684860823bb976e508bd71 P 9cf28505064b4b4ba7581338c203ee66: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf28505064b4b4ba7581338c203ee66" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38069 } } peers { permanent_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37269 } } peers { permanent_uuid: "49a7b8cf178d46f28c018c0bda345304" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45795 } } }
I20260926 16:30:50.814261 8523 tablet_replica.cc:442] T 32a51e7e22684860823bb976e508bd71 P 9cf28505064b4b4ba7581338c203ee66: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:30:50.814276 8519 raft_consensus.cc:348] T 32a51e7e22684860823bb976e508bd71 P 6a4cdd1c228144b1a0a7bd99cab216cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf28505064b4b4ba7581338c203ee66" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38069 } } peers { permanent_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37269 } } peers { permanent_uuid: "49a7b8cf178d46f28c018c0bda345304" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45795 } }
I20260926 16:30:50.814528 8519 raft_consensus.cc:374] T 32a51e7e22684860823bb976e508bd71 P 6a4cdd1c228144b1a0a7bd99cab216cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:30:50.814630 8519 raft_consensus.cc:729] T 32a51e7e22684860823bb976e508bd71 P 6a4cdd1c228144b1a0a7bd99cab216cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a4cdd1c228144b1a0a7bd99cab216cf, State: Initialized, Role: FOLLOWER
I20260926 16:30:50.814862 8519 consensus_queue.cc:260] T 32a51e7e22684860823bb976e508bd71 P 6a4cdd1c228144b1a0a7bd99cab216cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf28505064b4b4ba7581338c203ee66" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38069 } } peers { permanent_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37269 } } peers { permanent_uuid: "49a7b8cf178d46f28c018c0bda345304" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45795 } }
I20260926 16:30:50.815339 8525 tablet_replica.cc:439] T 32a51e7e22684860823bb976e508bd71 P 6a4cdd1c228144b1a0a7bd99cab216cf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf28505064b4b4ba7581338c203ee66" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38069 } } peers { permanent_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37269 } } peers { permanent_uuid: "49a7b8cf178d46f28c018c0bda345304" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45795 } } }
I20260926 16:30:50.815644 8525 tablet_replica.cc:442] T 32a51e7e22684860823bb976e508bd71 P 6a4cdd1c228144b1a0a7bd99cab216cf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:30:50.816658 8519 ts_tablet_manager.cc:1428] T 32a51e7e22684860823bb976e508bd71 P 6a4cdd1c228144b1a0a7bd99cab216cf: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260926 16:30:50.872952 8525 raft_consensus.cc:482] T 32a51e7e22684860823bb976e508bd71 P 6a4cdd1c228144b1a0a7bd99cab216cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:30:50.873219 8525 raft_consensus.cc:504] T 32a51e7e22684860823bb976e508bd71 P 6a4cdd1c228144b1a0a7bd99cab216cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf28505064b4b4ba7581338c203ee66" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38069 } } peers { permanent_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37269 } } peers { permanent_uuid: "49a7b8cf178d46f28c018c0bda345304" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45795 } }
I20260926 16:30:50.873884 8525 leader_election.cc:288] T 32a51e7e22684860823bb976e508bd71 P 6a4cdd1c228144b1a0a7bd99cab216cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9cf28505064b4b4ba7581338c203ee66 (127.6.33.1:38069), 49a7b8cf178d46f28c018c0bda345304 (127.6.33.3:45795)
I20260926 16:30:50.874490 8170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32a51e7e22684860823bb976e508bd71" candidate_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cf28505064b4b4ba7581338c203ee66" is_pre_election: true
I20260926 16:30:50.874456 8448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32a51e7e22684860823bb976e508bd71" candidate_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49a7b8cf178d46f28c018c0bda345304" is_pre_election: true
I20260926 16:30:50.874717 8170 raft_consensus.cc:2454] T 32a51e7e22684860823bb976e508bd71 P 9cf28505064b4b4ba7581338c203ee66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a4cdd1c228144b1a0a7bd99cab216cf in term 0.
I20260926 16:30:50.874742 8448 raft_consensus.cc:2454] T 32a51e7e22684860823bb976e508bd71 P 49a7b8cf178d46f28c018c0bda345304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a4cdd1c228144b1a0a7bd99cab216cf in term 0.
I20260926 16:30:50.875131 8242 leader_election.cc:302] T 32a51e7e22684860823bb976e508bd71 P 6a4cdd1c228144b1a0a7bd99cab216cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a4cdd1c228144b1a0a7bd99cab216cf, 9cf28505064b4b4ba7581338c203ee66; no voters:
I20260926 16:30:50.875438 8525 raft_consensus.cc:2781] T 32a51e7e22684860823bb976e508bd71 P 6a4cdd1c228144b1a0a7bd99cab216cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:30:50.875521 8525 raft_consensus.cc:482] T 32a51e7e22684860823bb976e508bd71 P 6a4cdd1c228144b1a0a7bd99cab216cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:30:50.875587 8525 raft_consensus.cc:3037] T 32a51e7e22684860823bb976e508bd71 P 6a4cdd1c228144b1a0a7bd99cab216cf [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:30:50.912555 8525 raft_consensus.cc:504] T 32a51e7e22684860823bb976e508bd71 P 6a4cdd1c228144b1a0a7bd99cab216cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf28505064b4b4ba7581338c203ee66" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38069 } } peers { permanent_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37269 } } peers { permanent_uuid: "49a7b8cf178d46f28c018c0bda345304" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45795 } }
I20260926 16:30:50.913435 8525 leader_election.cc:288] T 32a51e7e22684860823bb976e508bd71 P 6a4cdd1c228144b1a0a7bd99cab216cf [CANDIDATE]: Term 1 election: Requested vote from peers 9cf28505064b4b4ba7581338c203ee66 (127.6.33.1:38069), 49a7b8cf178d46f28c018c0bda345304 (127.6.33.3:45795)
I20260926 16:30:50.913991 8448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32a51e7e22684860823bb976e508bd71" candidate_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49a7b8cf178d46f28c018c0bda345304"
I20260926 16:30:50.914026 8170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32a51e7e22684860823bb976e508bd71" candidate_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cf28505064b4b4ba7581338c203ee66"
I20260926 16:30:50.914228 8448 raft_consensus.cc:3037] T 32a51e7e22684860823bb976e508bd71 P 49a7b8cf178d46f28c018c0bda345304 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:30:50.914296 8170 raft_consensus.cc:3037] T 32a51e7e22684860823bb976e508bd71 P 9cf28505064b4b4ba7581338c203ee66 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:30:50.916268 8448 raft_consensus.cc:2454] T 32a51e7e22684860823bb976e508bd71 P 49a7b8cf178d46f28c018c0bda345304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a4cdd1c228144b1a0a7bd99cab216cf in term 1.
I20260926 16:30:50.916568 8170 raft_consensus.cc:2454] T 32a51e7e22684860823bb976e508bd71 P 9cf28505064b4b4ba7581338c203ee66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a4cdd1c228144b1a0a7bd99cab216cf in term 1.
I20260926 16:30:50.916828 8240 leader_election.cc:302] T 32a51e7e22684860823bb976e508bd71 P 6a4cdd1c228144b1a0a7bd99cab216cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49a7b8cf178d46f28c018c0bda345304, 6a4cdd1c228144b1a0a7bd99cab216cf; no voters:
I20260926 16:30:50.917181 8525 raft_consensus.cc:2781] T 32a51e7e22684860823bb976e508bd71 P 6a4cdd1c228144b1a0a7bd99cab216cf [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:30:50.917634 8525 raft_consensus.cc:686] T 32a51e7e22684860823bb976e508bd71 P 6a4cdd1c228144b1a0a7bd99cab216cf [term 1 LEADER]: Becoming Leader. State: Replica: 6a4cdd1c228144b1a0a7bd99cab216cf, State: Running, Role: LEADER
I20260926 16:30:50.918213 8525 consensus_queue.cc:237] T 32a51e7e22684860823bb976e508bd71 P 6a4cdd1c228144b1a0a7bd99cab216cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf28505064b4b4ba7581338c203ee66" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38069 } } peers { permanent_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37269 } } peers { permanent_uuid: "49a7b8cf178d46f28c018c0bda345304" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45795 } }
I20260926 16:30:50.920848 8541 tablet_replica.cc:439] T 32a51e7e22684860823bb976e508bd71 P 6a4cdd1c228144b1a0a7bd99cab216cf: TxnStatusTablet state changed. Reason: New leader 6a4cdd1c228144b1a0a7bd99cab216cf. Latest consensus state: current_term: 1 leader_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf28505064b4b4ba7581338c203ee66" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38069 } } peers { permanent_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37269 } } peers { permanent_uuid: "49a7b8cf178d46f28c018c0bda345304" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45795 } } }
I20260926 16:30:50.921254 8541 tablet_replica.cc:442] T 32a51e7e22684860823bb976e508bd71 P 6a4cdd1c228144b1a0a7bd99cab216cf: This TxnStatusTablet replica's current role is: LEADER
I20260926 16:30:50.921998 8543 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 16:30:50.922297 8543 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 16:30:50.922933 8012 catalog_manager.cc:5495] T 32a51e7e22684860823bb976e508bd71 P 6a4cdd1c228144b1a0a7bd99cab216cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a4cdd1c228144b1a0a7bd99cab216cf (127.6.33.2). New cstate: current_term: 1 leader_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf28505064b4b4ba7581338c203ee66" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49a7b8cf178d46f28c018c0bda345304" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45795 } health_report { overall_health: UNKNOWN } } }
I20260926 16:30:50.943081 8537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:50.936429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.2:37269 (local address 127.0.0.1:51134)
0926 16:30:50.936818 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0926 16:30:50.936843 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 16:30:50.936968 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:30:50.937506 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:30:50.937511 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:30:50.937530 (+ 19us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 16:30:50.937716 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:50.937725 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:50.938817 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:50.938826 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:50.940590 (+ 1764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:50.940602 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:50.941431 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:50.941450 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:50.941608 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:50.941670 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 16:30:50.942543 (+ 873us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 16:30:50.942566 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 16:30:50.942641 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 16:30:50.942807 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209}
I20260926 16:30:50.943084 8513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:50.936420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51134 (local address 127.6.33.2:37269)
0926 16:30:50.936594 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:50.936600 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:50.936901 (+ 301us) server_negotiation.cc:407] Connection header received
0926 16:30:50.937194 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:50.937219 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:50.937337 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:50.937494 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:30:50.937846 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:50.938690 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:50.940796 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:50.941268 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:50.941403 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:50.941894 (+ 491us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:30:50.942428 (+ 534us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:30:50.942596 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:50.942659 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 16:30:50.942739 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 16:30:50.951053 8170 raft_consensus.cc:1261] T 32a51e7e22684860823bb976e508bd71 P 9cf28505064b4b4ba7581338c203ee66 [term 1 FOLLOWER]: Refusing update from remote peer 6a4cdd1c228144b1a0a7bd99cab216cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:30:50.951138 8448 raft_consensus.cc:1261] T 32a51e7e22684860823bb976e508bd71 P 49a7b8cf178d46f28c018c0bda345304 [term 1 FOLLOWER]: Refusing update from remote peer 6a4cdd1c228144b1a0a7bd99cab216cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:30:50.952016 8525 consensus_queue.cc:1035] T 32a51e7e22684860823bb976e508bd71 P 6a4cdd1c228144b1a0a7bd99cab216cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf28505064b4b4ba7581338c203ee66" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:30:50.952358 8541 consensus_queue.cc:1035] T 32a51e7e22684860823bb976e508bd71 P 6a4cdd1c228144b1a0a7bd99cab216cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "49a7b8cf178d46f28c018c0bda345304" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:30:50.969240 8524 tablet_replica.cc:439] T 32a51e7e22684860823bb976e508bd71 P 49a7b8cf178d46f28c018c0bda345304: TxnStatusTablet state changed. Reason: New leader 6a4cdd1c228144b1a0a7bd99cab216cf. Latest consensus state: current_term: 1 leader_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf28505064b4b4ba7581338c203ee66" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38069 } } peers { permanent_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37269 } } peers { permanent_uuid: "49a7b8cf178d46f28c018c0bda345304" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45795 } } }
I20260926 16:30:50.969556 8524 tablet_replica.cc:442] T 32a51e7e22684860823bb976e508bd71 P 49a7b8cf178d46f28c018c0bda345304: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:30:50.969913 8523 tablet_replica.cc:439] T 32a51e7e22684860823bb976e508bd71 P 9cf28505064b4b4ba7581338c203ee66: TxnStatusTablet state changed. Reason: New leader 6a4cdd1c228144b1a0a7bd99cab216cf. Latest consensus state: current_term: 1 leader_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf28505064b4b4ba7581338c203ee66" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38069 } } peers { permanent_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37269 } } peers { permanent_uuid: "49a7b8cf178d46f28c018c0bda345304" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45795 } } }
I20260926 16:30:50.970234 8523 tablet_replica.cc:442] T 32a51e7e22684860823bb976e508bd71 P 9cf28505064b4b4ba7581338c203ee66: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:30:50.973147 8546 mvcc.cc:205] Tried to move back new op lower bound from 7333643267885928448 to 7333643267764846592. Current Snapshot: MvccSnapshot[applied={T|T < 7333643267885928448}]
I20260926 16:30:50.977174 8548 mvcc.cc:205] Tried to move back new op lower bound from 7333643267885928448 to 7333643267764846592. Current Snapshot: MvccSnapshot[applied={T|T < 7333643267885928448}]
I20260926 16:30:50.977867 8541 tablet_replica.cc:439] T 32a51e7e22684860823bb976e508bd71 P 6a4cdd1c228144b1a0a7bd99cab216cf: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf28505064b4b4ba7581338c203ee66" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38069 } } peers { permanent_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37269 } } peers { permanent_uuid: "49a7b8cf178d46f28c018c0bda345304" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45795 } } }
I20260926 16:30:50.978122 8541 tablet_replica.cc:442] T 32a51e7e22684860823bb976e508bd71 P 6a4cdd1c228144b1a0a7bd99cab216cf: This TxnStatusTablet replica's current role is: LEADER
I20260926 16:30:50.979274 8541 tablet_replica.cc:439] T 32a51e7e22684860823bb976e508bd71 P 6a4cdd1c228144b1a0a7bd99cab216cf: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf28505064b4b4ba7581338c203ee66" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38069 } } peers { permanent_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37269 } } peers { permanent_uuid: "49a7b8cf178d46f28c018c0bda345304" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45795 } } }
I20260926 16:30:50.979518 8541 tablet_replica.cc:442] T 32a51e7e22684860823bb976e508bd71 P 6a4cdd1c228144b1a0a7bd99cab216cf: This TxnStatusTablet replica's current role is: LEADER
I20260926 16:30:50.981105 8524 tablet_replica.cc:439] T 32a51e7e22684860823bb976e508bd71 P 49a7b8cf178d46f28c018c0bda345304: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf28505064b4b4ba7581338c203ee66" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38069 } } peers { permanent_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37269 } } peers { permanent_uuid: "49a7b8cf178d46f28c018c0bda345304" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45795 } } }
I20260926 16:30:50.981325 8524 tablet_replica.cc:442] T 32a51e7e22684860823bb976e508bd71 P 49a7b8cf178d46f28c018c0bda345304: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:30:50.986157 8523 tablet_replica.cc:439] T 32a51e7e22684860823bb976e508bd71 P 9cf28505064b4b4ba7581338c203ee66: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf28505064b4b4ba7581338c203ee66" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38069 } } peers { permanent_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37269 } } peers { permanent_uuid: "49a7b8cf178d46f28c018c0bda345304" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45795 } } }
I20260926 16:30:50.986467 8523 tablet_replica.cc:442] T 32a51e7e22684860823bb976e508bd71 P 9cf28505064b4b4ba7581338c203ee66: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:30:50.995230 8516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:50.988948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50026 (local address 127.6.33.1:38069)
0926 16:30:50.989132 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:50.989141 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:50.989673 (+ 532us) server_negotiation.cc:407] Connection header received
0926 16:30:50.989972 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:50.989977 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:50.990078 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:50.990273 (+ 195us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:30:50.990703 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:50.991593 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:50.993120 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:50.993515 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:50.993650 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:50.994058 (+ 408us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:30:50.994613 (+ 555us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:30:50.994761 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:50.994856 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 16:30:50.994931 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 16:30:51.019655 8513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:51.013229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:44631 (local address 127.6.33.2:37269)
0926 16:30:51.013437 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:51.013444 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:51.014024 (+ 580us) server_negotiation.cc:407] Connection header received
0926 16:30:51.014239 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:51.014243 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:51.014313 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:51.014390 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:30:51.014970 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:51.015751 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:51.017889 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:51.018381 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:51.018489 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:51.019056 (+ 567us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:30:51.019156 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:30:51.019238 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:51.019382 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 16:30:51.019446 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 16:30:51.019898 8558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:51.013205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.2:37269 (local address 127.6.33.1:44631)
0926 16:30:51.013906 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0926 16:30:51.013932 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 16:30:51.014087 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:30:51.014433 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:30:51.014438 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:30:51.014451 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 16:30:51.014819 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:51.014830 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:51.015885 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:51.015891 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:51.017739 (+ 1848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:51.017755 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:51.018564 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:51.018571 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:51.018774 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:51.018852 (+ 78us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 16:30:51.019262 (+ 410us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 16:30:51.019276 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 16:30:51.019357 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 16:30:51.019511 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":366,"threads_started":1}
I20260926 16:30:51.029956 8448 raft_consensus.cc:1261] T 1dc7e5c0faee4950ad0e763903a8084e P 49a7b8cf178d46f28c018c0bda345304 [term 1 FOLLOWER]: Refusing update from remote peer 9cf28505064b4b4ba7581338c203ee66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:30:51.030385 8309 raft_consensus.cc:1261] T 1dc7e5c0faee4950ad0e763903a8084e P 6a4cdd1c228144b1a0a7bd99cab216cf [term 1 FOLLOWER]: Refusing update from remote peer 9cf28505064b4b4ba7581338c203ee66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:30:51.030609 8530 consensus_queue.cc:1035] T 1dc7e5c0faee4950ad0e763903a8084e P 9cf28505064b4b4ba7581338c203ee66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49a7b8cf178d46f28c018c0bda345304" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:30:51.031076 8523 consensus_queue.cc:1035] T 1dc7e5c0faee4950ad0e763903a8084e P 9cf28505064b4b4ba7581338c203ee66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:30:51.077409 8516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:51.071068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:45945 (local address 127.6.33.1:38069)
0926 16:30:51.071263 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:51.071272 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:51.071759 (+ 487us) server_negotiation.cc:407] Connection header received
0926 16:30:51.071985 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:51.071996 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:51.072056 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:51.072151 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:30:51.072743 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:51.073825 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:51.075409 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:51.076006 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:51.076140 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:51.076594 (+ 454us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:30:51.076684 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:30:51.076738 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:51.077019 (+ 281us) server_negotiation.cc:299] Negotiation successful
0926 16:30:51.077108 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 16:30:51.078102 8565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:51.071040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:38069 (local address 127.6.33.2:45945)
0926 16:30:51.071636 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0926 16:30:51.071651 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 16:30:51.071844 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:30:51.072406 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:30:51.072410 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:30:51.072419 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 16:30:51.072599 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:51.072606 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:51.073945 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:51.073952 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:51.075227 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:51.075239 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:51.076150 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:51.076155 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:51.076326 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:51.076402 (+ 76us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 16:30:51.076859 (+ 457us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 16:30:51.076873 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 16:30:51.077766 (+ 893us) client_negotiation.cc:240] Negotiation successful
0926 16:30:51.077873 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":288,"threads_started":1}
I20260926 16:30:51.118426 8011 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:30:51.118798 8011 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:30:51.122819 8011 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4ed326500513428ebd1beeff75c87965: Sending DeleteTablet for 3 replicas of tablet 1dc7e5c0faee4950ad0e763903a8084e
I20260926 16:30:51.123900 8150 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dc7e5c0faee4950ad0e763903a8084e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:30:51 UTC) from {username='kudu', principal='kudu/127.6.33.62@KRBTEST.COM'} at 127.0.0.1:50020
I20260926 16:30:51.123991 8288 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dc7e5c0faee4950ad0e763903a8084e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:30:51 UTC) from {username='kudu', principal='kudu/127.6.33.62@KRBTEST.COM'} at 127.0.0.1:51128
I20260926 16:30:51.125447 8428 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dc7e5c0faee4950ad0e763903a8084e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:30:51 UTC) from {username='kudu', principal='kudu/127.6.33.62@KRBTEST.COM'} at 127.0.0.1:34310
I20260926 16:30:51.126479 8011 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52212:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:30:51.127493 8011 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:30:51.139580 8149 tablet_service.cc:1467] Processing CreateTablet for tablet 21fc895b92974a7586d46e85d951bda2 (DEFAULT_TABLE table=test-table [id=572c064ca6a948db8b9e9ab257e63c3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:30:51.140340 8149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21fc895b92974a7586d46e85d951bda2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:30:51.146881 8289 tablet_service.cc:1467] Processing CreateTablet for tablet 21fc895b92974a7586d46e85d951bda2 (DEFAULT_TABLE table=test-table [id=572c064ca6a948db8b9e9ab257e63c3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:30:51.147379 8289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21fc895b92974a7586d46e85d951bda2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:30:51.149236 8427 tablet_service.cc:1467] Processing CreateTablet for tablet 21fc895b92974a7586d46e85d951bda2 (DEFAULT_TABLE table=test-table [id=572c064ca6a948db8b9e9ab257e63c3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:30:51.149689 8427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21fc895b92974a7586d46e85d951bda2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:30:51.153380 8517 tablet_bootstrap.cc:492] T 21fc895b92974a7586d46e85d951bda2 P 9cf28505064b4b4ba7581338c203ee66: Bootstrap starting.
I20260926 16:30:51.155167 8517 tablet_bootstrap.cc:654] T 21fc895b92974a7586d46e85d951bda2 P 9cf28505064b4b4ba7581338c203ee66: Neither blocks nor log segments found. Creating new log.
I20260926 16:30:51.160950 8519 tablet_bootstrap.cc:492] T 21fc895b92974a7586d46e85d951bda2 P 6a4cdd1c228144b1a0a7bd99cab216cf: Bootstrap starting.
I20260926 16:30:51.162887 8519 tablet_bootstrap.cc:654] T 21fc895b92974a7586d46e85d951bda2 P 6a4cdd1c228144b1a0a7bd99cab216cf: Neither blocks nor log segments found. Creating new log.
I20260926 16:30:51.165676 8518 tablet_bootstrap.cc:492] T 21fc895b92974a7586d46e85d951bda2 P 49a7b8cf178d46f28c018c0bda345304: Bootstrap starting.
I20260926 16:30:51.167886 8518 tablet_bootstrap.cc:654] T 21fc895b92974a7586d46e85d951bda2 P 49a7b8cf178d46f28c018c0bda345304: Neither blocks nor log segments found. Creating new log.
I20260926 16:30:51.182490 8569 tablet_replica.cc:330] T 1dc7e5c0faee4950ad0e763903a8084e P 6a4cdd1c228144b1a0a7bd99cab216cf: stopping tablet replica
I20260926 16:30:51.183086 8569 raft_consensus.cc:2229] T 1dc7e5c0faee4950ad0e763903a8084e P 6a4cdd1c228144b1a0a7bd99cab216cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:30:51.183596 8569 raft_consensus.cc:2258] T 1dc7e5c0faee4950ad0e763903a8084e P 6a4cdd1c228144b1a0a7bd99cab216cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:30:51.186369 8570 tablet_replica.cc:330] T 1dc7e5c0faee4950ad0e763903a8084e P 49a7b8cf178d46f28c018c0bda345304: stopping tablet replica
I20260926 16:30:51.186820 8570 raft_consensus.cc:2229] T 1dc7e5c0faee4950ad0e763903a8084e P 49a7b8cf178d46f28c018c0bda345304 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:30:51.187251 8570 raft_consensus.cc:2258] T 1dc7e5c0faee4950ad0e763903a8084e P 49a7b8cf178d46f28c018c0bda345304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:30:51.188587 8568 tablet_replica.cc:330] T 1dc7e5c0faee4950ad0e763903a8084e P 9cf28505064b4b4ba7581338c203ee66: stopping tablet replica
I20260926 16:30:51.189611 8569 ts_tablet_manager.cc:1905] T 1dc7e5c0faee4950ad0e763903a8084e P 6a4cdd1c228144b1a0a7bd99cab216cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:30:51.190702 8517 tablet_bootstrap.cc:492] T 21fc895b92974a7586d46e85d951bda2 P 9cf28505064b4b4ba7581338c203ee66: No bootstrap required, opened a new log
I20260926 16:30:51.190910 8517 ts_tablet_manager.cc:1397] T 21fc895b92974a7586d46e85d951bda2 P 9cf28505064b4b4ba7581338c203ee66: Time spent bootstrapping tablet: real 0.038s user 0.006s sys 0.004s
I20260926 16:30:51.191385 8568 raft_consensus.cc:2229] T 1dc7e5c0faee4950ad0e763903a8084e P 9cf28505064b4b4ba7581338c203ee66 [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:30:51.191821 8517 raft_consensus.cc:348] T 21fc895b92974a7586d46e85d951bda2 P 9cf28505064b4b4ba7581338c203ee66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf28505064b4b4ba7581338c203ee66" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38069 } } peers { permanent_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37269 } } peers { permanent_uuid: "49a7b8cf178d46f28c018c0bda345304" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45795 } }
I20260926 16:30:51.193719 8570 ts_tablet_manager.cc:1905] T 1dc7e5c0faee4950ad0e763903a8084e P 49a7b8cf178d46f28c018c0bda345304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:30:51.205404 8569 ts_tablet_manager.cc:1918] T 1dc7e5c0faee4950ad0e763903a8084e P 6a4cdd1c228144b1a0a7bd99cab216cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 16:30:51.207022 8569 log.cc:1196] T 1dc7e5c0faee4950ad0e763903a8084e P 6a4cdd1c228144b1a0a7bd99cab216cf: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/ts-1/wal/wals/1dc7e5c0faee4950ad0e763903a8084e
I20260926 16:30:51.207132 8517 raft_consensus.cc:374] T 21fc895b92974a7586d46e85d951bda2 P 9cf28505064b4b4ba7581338c203ee66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:30:51.207273 8517 raft_consensus.cc:729] T 21fc895b92974a7586d46e85d951bda2 P 9cf28505064b4b4ba7581338c203ee66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cf28505064b4b4ba7581338c203ee66, State: Initialized, Role: FOLLOWER
I20260926 16:30:51.207280 8568 raft_consensus.cc:2258] T 1dc7e5c0faee4950ad0e763903a8084e P 9cf28505064b4b4ba7581338c203ee66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:30:51.207837 8517 consensus_queue.cc:260] T 21fc895b92974a7586d46e85d951bda2 P 9cf28505064b4b4ba7581338c203ee66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf28505064b4b4ba7581338c203ee66" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38069 } } peers { permanent_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37269 } } peers { permanent_uuid: "49a7b8cf178d46f28c018c0bda345304" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45795 } }
I20260926 16:30:51.208498 8517 ts_tablet_manager.cc:1428] T 21fc895b92974a7586d46e85d951bda2 P 9cf28505064b4b4ba7581338c203ee66: Time spent starting tablet: real 0.017s user 0.002s sys 0.001s
I20260926 16:30:51.208807 8518 tablet_bootstrap.cc:492] T 21fc895b92974a7586d46e85d951bda2 P 49a7b8cf178d46f28c018c0bda345304: No bootstrap required, opened a new log
I20260926 16:30:51.208977 8518 ts_tablet_manager.cc:1397] T 21fc895b92974a7586d46e85d951bda2 P 49a7b8cf178d46f28c018c0bda345304: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.007s
I20260926 16:30:51.209751 8518 raft_consensus.cc:348] T 21fc895b92974a7586d46e85d951bda2 P 49a7b8cf178d46f28c018c0bda345304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf28505064b4b4ba7581338c203ee66" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38069 } } peers { permanent_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37269 } } peers { permanent_uuid: "49a7b8cf178d46f28c018c0bda345304" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45795 } }
I20260926 16:30:51.210057 8518 raft_consensus.cc:374] T 21fc895b92974a7586d46e85d951bda2 P 49a7b8cf178d46f28c018c0bda345304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:30:51.210171 8518 raft_consensus.cc:729] T 21fc895b92974a7586d46e85d951bda2 P 49a7b8cf178d46f28c018c0bda345304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49a7b8cf178d46f28c018c0bda345304, State: Initialized, Role: FOLLOWER
I20260926 16:30:51.210493 8518 consensus_queue.cc:260] T 21fc895b92974a7586d46e85d951bda2 P 49a7b8cf178d46f28c018c0bda345304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf28505064b4b4ba7581338c203ee66" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38069 } } peers { permanent_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37269 } } peers { permanent_uuid: "49a7b8cf178d46f28c018c0bda345304" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45795 } }
I20260926 16:30:51.210743 8568 ts_tablet_manager.cc:1905] T 1dc7e5c0faee4950ad0e763903a8084e P 9cf28505064b4b4ba7581338c203ee66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:30:51.215726 8518 ts_tablet_manager.cc:1428] T 21fc895b92974a7586d46e85d951bda2 P 49a7b8cf178d46f28c018c0bda345304: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20260926 16:30:51.221876 8568 ts_tablet_manager.cc:1918] T 1dc7e5c0faee4950ad0e763903a8084e P 9cf28505064b4b4ba7581338c203ee66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 16:30:51.217550 8519 tablet_bootstrap.cc:492] T 21fc895b92974a7586d46e85d951bda2 P 6a4cdd1c228144b1a0a7bd99cab216cf: No bootstrap required, opened a new log
I20260926 16:30:51.222328 8519 ts_tablet_manager.cc:1397] T 21fc895b92974a7586d46e85d951bda2 P 6a4cdd1c228144b1a0a7bd99cab216cf: Time spent bootstrapping tablet: real 0.061s user 0.004s sys 0.016s
I20260926 16:30:51.222883 8569 ts_tablet_manager.cc:1939] T 1dc7e5c0faee4950ad0e763903a8084e P 6a4cdd1c228144b1a0a7bd99cab216cf: Deleting consensus metadata
I20260926 16:30:51.223240 8519 raft_consensus.cc:348] T 21fc895b92974a7586d46e85d951bda2 P 6a4cdd1c228144b1a0a7bd99cab216cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf28505064b4b4ba7581338c203ee66" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38069 } } peers { permanent_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37269 } } peers { permanent_uuid: "49a7b8cf178d46f28c018c0bda345304" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45795 } }
I20260926 16:30:51.223564 8519 raft_consensus.cc:374] T 21fc895b92974a7586d46e85d951bda2 P 6a4cdd1c228144b1a0a7bd99cab216cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:30:51.223681 8519 raft_consensus.cc:729] T 21fc895b92974a7586d46e85d951bda2 P 6a4cdd1c228144b1a0a7bd99cab216cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a4cdd1c228144b1a0a7bd99cab216cf, State: Initialized, Role: FOLLOWER
I20260926 16:30:51.224030 8519 consensus_queue.cc:260] T 21fc895b92974a7586d46e85d951bda2 P 6a4cdd1c228144b1a0a7bd99cab216cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf28505064b4b4ba7581338c203ee66" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38069 } } peers { permanent_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37269 } } peers { permanent_uuid: "49a7b8cf178d46f28c018c0bda345304" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45795 } }
I20260926 16:30:51.224818 8000 catalog_manager.cc:4841] TS 6a4cdd1c228144b1a0a7bd99cab216cf (127.6.33.2:37269): tablet 1dc7e5c0faee4950ad0e763903a8084e (table test-table [id=e2767078a9d340e4943f5cc73ad92036]) successfully deleted
I20260926 16:30:51.225591 8519 ts_tablet_manager.cc:1428] T 21fc895b92974a7586d46e85d951bda2 P 6a4cdd1c228144b1a0a7bd99cab216cf: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260926 16:30:51.228523 8568 log.cc:1196] T 1dc7e5c0faee4950ad0e763903a8084e P 9cf28505064b4b4ba7581338c203ee66: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/ts-0/wal/wals/1dc7e5c0faee4950ad0e763903a8084e
I20260926 16:30:51.229179 8570 ts_tablet_manager.cc:1918] T 1dc7e5c0faee4950ad0e763903a8084e P 49a7b8cf178d46f28c018c0bda345304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 16:30:51.229266 8568 ts_tablet_manager.cc:1939] T 1dc7e5c0faee4950ad0e763903a8084e P 9cf28505064b4b4ba7581338c203ee66: Deleting consensus metadata
I20260926 16:30:51.229396 8570 log.cc:1196] T 1dc7e5c0faee4950ad0e763903a8084e P 49a7b8cf178d46f28c018c0bda345304: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/ts-2/wal/wals/1dc7e5c0faee4950ad0e763903a8084e
I20260926 16:30:51.229903 8570 ts_tablet_manager.cc:1939] T 1dc7e5c0faee4950ad0e763903a8084e P 49a7b8cf178d46f28c018c0bda345304: Deleting consensus metadata
I20260926 16:30:51.231572 7997 catalog_manager.cc:4841] TS 49a7b8cf178d46f28c018c0bda345304 (127.6.33.3:45795): tablet 1dc7e5c0faee4950ad0e763903a8084e (table test-table [id=e2767078a9d340e4943f5cc73ad92036]) successfully deleted
I20260926 16:30:51.232252 7999 catalog_manager.cc:4841] TS 9cf28505064b4b4ba7581338c203ee66 (127.6.33.1:38069): tablet 1dc7e5c0faee4950ad0e763903a8084e (table test-table [id=e2767078a9d340e4943f5cc73ad92036]) successfully deleted
I20260926 16:30:51.277558 8525 raft_consensus.cc:482] T 21fc895b92974a7586d46e85d951bda2 P 6a4cdd1c228144b1a0a7bd99cab216cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:30:51.277837 8525 raft_consensus.cc:504] T 21fc895b92974a7586d46e85d951bda2 P 6a4cdd1c228144b1a0a7bd99cab216cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf28505064b4b4ba7581338c203ee66" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38069 } } peers { permanent_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37269 } } peers { permanent_uuid: "49a7b8cf178d46f28c018c0bda345304" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45795 } }
I20260926 16:30:51.278656 8525 leader_election.cc:288] T 21fc895b92974a7586d46e85d951bda2 P 6a4cdd1c228144b1a0a7bd99cab216cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9cf28505064b4b4ba7581338c203ee66 (127.6.33.1:38069), 49a7b8cf178d46f28c018c0bda345304 (127.6.33.3:45795)
I20260926 16:30:51.279368 8448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21fc895b92974a7586d46e85d951bda2" candidate_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49a7b8cf178d46f28c018c0bda345304" is_pre_election: true
I20260926 16:30:51.279373 8170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21fc895b92974a7586d46e85d951bda2" candidate_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cf28505064b4b4ba7581338c203ee66" is_pre_election: true
I20260926 16:30:51.279598 8448 raft_consensus.cc:2454] T 21fc895b92974a7586d46e85d951bda2 P 49a7b8cf178d46f28c018c0bda345304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a4cdd1c228144b1a0a7bd99cab216cf in term 0.
I20260926 16:30:51.279637 8170 raft_consensus.cc:2454] T 21fc895b92974a7586d46e85d951bda2 P 9cf28505064b4b4ba7581338c203ee66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a4cdd1c228144b1a0a7bd99cab216cf in term 0.
I20260926 16:30:51.280251 8242 leader_election.cc:302] T 21fc895b92974a7586d46e85d951bda2 P 6a4cdd1c228144b1a0a7bd99cab216cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a4cdd1c228144b1a0a7bd99cab216cf, 9cf28505064b4b4ba7581338c203ee66; no voters:
I20260926 16:30:51.280587 8525 raft_consensus.cc:2781] T 21fc895b92974a7586d46e85d951bda2 P 6a4cdd1c228144b1a0a7bd99cab216cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:30:51.280696 8525 raft_consensus.cc:482] T 21fc895b92974a7586d46e85d951bda2 P 6a4cdd1c228144b1a0a7bd99cab216cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:30:51.280838 8525 raft_consensus.cc:3037] T 21fc895b92974a7586d46e85d951bda2 P 6a4cdd1c228144b1a0a7bd99cab216cf [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:30:51.284091 8525 raft_consensus.cc:504] T 21fc895b92974a7586d46e85d951bda2 P 6a4cdd1c228144b1a0a7bd99cab216cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf28505064b4b4ba7581338c203ee66" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38069 } } peers { permanent_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37269 } } peers { permanent_uuid: "49a7b8cf178d46f28c018c0bda345304" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45795 } }
I20260926 16:30:51.284900 8525 leader_election.cc:288] T 21fc895b92974a7586d46e85d951bda2 P 6a4cdd1c228144b1a0a7bd99cab216cf [CANDIDATE]: Term 1 election: Requested vote from peers 9cf28505064b4b4ba7581338c203ee66 (127.6.33.1:38069), 49a7b8cf178d46f28c018c0bda345304 (127.6.33.3:45795)
I20260926 16:30:51.285517 8448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21fc895b92974a7586d46e85d951bda2" candidate_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49a7b8cf178d46f28c018c0bda345304"
I20260926 16:30:51.285545 8170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21fc895b92974a7586d46e85d951bda2" candidate_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cf28505064b4b4ba7581338c203ee66"
I20260926 16:30:51.285749 8448 raft_consensus.cc:3037] T 21fc895b92974a7586d46e85d951bda2 P 49a7b8cf178d46f28c018c0bda345304 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:30:51.285755 8170 raft_consensus.cc:3037] T 21fc895b92974a7586d46e85d951bda2 P 9cf28505064b4b4ba7581338c203ee66 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:30:51.288664 8448 raft_consensus.cc:2454] T 21fc895b92974a7586d46e85d951bda2 P 49a7b8cf178d46f28c018c0bda345304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a4cdd1c228144b1a0a7bd99cab216cf in term 1.
I20260926 16:30:51.288995 8170 raft_consensus.cc:2454] T 21fc895b92974a7586d46e85d951bda2 P 9cf28505064b4b4ba7581338c203ee66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a4cdd1c228144b1a0a7bd99cab216cf in term 1.
I20260926 16:30:51.289407 8240 leader_election.cc:302] T 21fc895b92974a7586d46e85d951bda2 P 6a4cdd1c228144b1a0a7bd99cab216cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49a7b8cf178d46f28c018c0bda345304, 6a4cdd1c228144b1a0a7bd99cab216cf; no voters:
I20260926 16:30:51.289726 8525 raft_consensus.cc:2781] T 21fc895b92974a7586d46e85d951bda2 P 6a4cdd1c228144b1a0a7bd99cab216cf [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:30:51.289918 8525 raft_consensus.cc:686] T 21fc895b92974a7586d46e85d951bda2 P 6a4cdd1c228144b1a0a7bd99cab216cf [term 1 LEADER]: Becoming Leader. State: Replica: 6a4cdd1c228144b1a0a7bd99cab216cf, State: Running, Role: LEADER
I20260926 16:30:51.290285 8525 consensus_queue.cc:237] T 21fc895b92974a7586d46e85d951bda2 P 6a4cdd1c228144b1a0a7bd99cab216cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf28505064b4b4ba7581338c203ee66" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38069 } } peers { permanent_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37269 } } peers { permanent_uuid: "49a7b8cf178d46f28c018c0bda345304" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45795 } }
I20260926 16:30:51.294727 8011 catalog_manager.cc:5495] T 21fc895b92974a7586d46e85d951bda2 P 6a4cdd1c228144b1a0a7bd99cab216cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a4cdd1c228144b1a0a7bd99cab216cf (127.6.33.2). New cstate: current_term: 1 leader_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf28505064b4b4ba7581338c203ee66" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49a7b8cf178d46f28c018c0bda345304" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45795 } health_report { overall_health: UNKNOWN } } }
I20260926 16:30:51.345227 8513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:51.338198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51142 (local address 127.6.33.2:37269)
0926 16:30:51.338358 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:51.338365 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:51.338692 (+ 327us) server_negotiation.cc:407] Connection header received
0926 16:30:51.338879 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:51.338884 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:51.338975 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:51.339067 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:30:51.339528 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:51.340864 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:51.342915 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:51.343624 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:51.343764 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:51.344281 (+ 517us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:30:51.344418 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:30:51.344506 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:51.344772 (+ 266us) server_negotiation.cc:299] Negotiation successful
0926 16:30:51.344860 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 16:30:51.362197 8565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:51.355388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.2:37269 (local address 127.6.33.2:45147)
0926 16:30:51.355690 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0926 16:30:51.355727 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 16:30:51.355857 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:30:51.356248 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:30:51.356253 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:30:51.356265 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 16:30:51.356486 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:51.356500 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:51.357990 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:51.357996 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:51.359743 (+ 1747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:51.359759 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:51.360839 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:51.360846 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:51.361029 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:51.361075 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 16:30:51.361533 (+ 458us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 16:30:51.361544 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 16:30:51.361642 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 16:30:51.361807 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":4}
I20260926 16:30:51.362203 8513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:51.355340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:45147 (local address 127.6.33.2:37269)
0926 16:30:51.355514 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:51.355523 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:51.355806 (+ 283us) server_negotiation.cc:407] Connection header received
0926 16:30:51.355983 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:51.355989 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:51.356087 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:51.356183 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:30:51.356638 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:51.357833 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:51.359912 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:51.360622 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:51.360838 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:51.361265 (+ 427us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:30:51.361410 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:30:51.361509 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:51.361676 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 16:30:51.361824 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":73,"server-negotiator.queue_time_us":51}
I20260926 16:30:51.372370 8448 raft_consensus.cc:1261] T 21fc895b92974a7586d46e85d951bda2 P 49a7b8cf178d46f28c018c0bda345304 [term 1 FOLLOWER]: Refusing update from remote peer 6a4cdd1c228144b1a0a7bd99cab216cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:30:51.373188 8168 raft_consensus.cc:1261] T 21fc895b92974a7586d46e85d951bda2 P 9cf28505064b4b4ba7581338c203ee66 [term 1 FOLLOWER]: Refusing update from remote peer 6a4cdd1c228144b1a0a7bd99cab216cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:30:51.374176 8525 consensus_queue.cc:1035] T 21fc895b92974a7586d46e85d951bda2 P 6a4cdd1c228144b1a0a7bd99cab216cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "49a7b8cf178d46f28c018c0bda345304" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:30:51.374784 8562 consensus_queue.cc:1035] T 21fc895b92974a7586d46e85d951bda2 P 6a4cdd1c228144b1a0a7bd99cab216cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf28505064b4b4ba7581338c203ee66" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 16:30:51.414634 8561 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 16:30:53.027241 8012 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:30:53.027498 8012 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:30:53.032004 8012 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4ed326500513428ebd1beeff75c87965: Sending DeleteTablet for 3 replicas of tablet 21fc895b92974a7586d46e85d951bda2
I20260926 16:30:53.032876 8150 tablet_service.cc:1514] Processing DeleteTablet for tablet 21fc895b92974a7586d46e85d951bda2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:30:53 UTC) from {username='kudu', principal='kudu/127.6.33.62@KRBTEST.COM'} at 127.0.0.1:50020
I20260926 16:30:53.032887 8289 tablet_service.cc:1514] Processing DeleteTablet for tablet 21fc895b92974a7586d46e85d951bda2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:30:53 UTC) from {username='kudu', principal='kudu/127.6.33.62@KRBTEST.COM'} at 127.0.0.1:51128
I20260926 16:30:53.032966 8428 tablet_service.cc:1514] Processing DeleteTablet for tablet 21fc895b92974a7586d46e85d951bda2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:30:53 UTC) from {username='kudu', principal='kudu/127.6.33.62@KRBTEST.COM'} at 127.0.0.1:34310
I20260926 16:30:53.045630 8599 tablet_replica.cc:330] T 21fc895b92974a7586d46e85d951bda2 P 9cf28505064b4b4ba7581338c203ee66: stopping tablet replica
I20260926 16:30:53.045953 8598 tablet_replica.cc:330] T 21fc895b92974a7586d46e85d951bda2 P 6a4cdd1c228144b1a0a7bd99cab216cf: stopping tablet replica
I20260926 16:30:53.046120 8600 tablet_replica.cc:330] T 21fc895b92974a7586d46e85d951bda2 P 49a7b8cf178d46f28c018c0bda345304: stopping tablet replica
I20260926 16:30:53.046290 8598 raft_consensus.cc:2229] T 21fc895b92974a7586d46e85d951bda2 P 6a4cdd1c228144b1a0a7bd99cab216cf [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:30:53.046301 8600 raft_consensus.cc:2229] T 21fc895b92974a7586d46e85d951bda2 P 49a7b8cf178d46f28c018c0bda345304 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:30:53.046554 8600 raft_consensus.cc:2258] T 21fc895b92974a7586d46e85d951bda2 P 49a7b8cf178d46f28c018c0bda345304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:30:53.046684 8599 raft_consensus.cc:2229] T 21fc895b92974a7586d46e85d951bda2 P 9cf28505064b4b4ba7581338c203ee66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:30:53.046933 8599 raft_consensus.cc:2258] T 21fc895b92974a7586d46e85d951bda2 P 9cf28505064b4b4ba7581338c203ee66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:30:53.046931 8598 raft_consensus.cc:2258] T 21fc895b92974a7586d46e85d951bda2 P 6a4cdd1c228144b1a0a7bd99cab216cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:30:53.047894 8600 ts_tablet_manager.cc:1905] T 21fc895b92974a7586d46e85d951bda2 P 49a7b8cf178d46f28c018c0bda345304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:30:53.047894 8598 ts_tablet_manager.cc:1905] T 21fc895b92974a7586d46e85d951bda2 P 6a4cdd1c228144b1a0a7bd99cab216cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:30:53.047935 8599 ts_tablet_manager.cc:1905] T 21fc895b92974a7586d46e85d951bda2 P 9cf28505064b4b4ba7581338c203ee66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:30:53.054854 8598 ts_tablet_manager.cc:1918] T 21fc895b92974a7586d46e85d951bda2 P 6a4cdd1c228144b1a0a7bd99cab216cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 16:30:53.054863 8600 ts_tablet_manager.cc:1918] T 21fc895b92974a7586d46e85d951bda2 P 49a7b8cf178d46f28c018c0bda345304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 16:30:53.055012 8600 log.cc:1196] T 21fc895b92974a7586d46e85d951bda2 P 49a7b8cf178d46f28c018c0bda345304: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/ts-2/wal/wals/21fc895b92974a7586d46e85d951bda2
I20260926 16:30:53.055119 8598 log.cc:1196] T 21fc895b92974a7586d46e85d951bda2 P 6a4cdd1c228144b1a0a7bd99cab216cf: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/ts-1/wal/wals/21fc895b92974a7586d46e85d951bda2
I20260926 16:30:53.055466 8599 ts_tablet_manager.cc:1918] T 21fc895b92974a7586d46e85d951bda2 P 9cf28505064b4b4ba7581338c203ee66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 16:30:53.055572 8600 ts_tablet_manager.cc:1939] T 21fc895b92974a7586d46e85d951bda2 P 49a7b8cf178d46f28c018c0bda345304: Deleting consensus metadata
I20260926 16:30:53.055663 8599 log.cc:1196] T 21fc895b92974a7586d46e85d951bda2 P 9cf28505064b4b4ba7581338c203ee66: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/ts-0/wal/wals/21fc895b92974a7586d46e85d951bda2
I20260926 16:30:53.055701 8598 ts_tablet_manager.cc:1939] T 21fc895b92974a7586d46e85d951bda2 P 6a4cdd1c228144b1a0a7bd99cab216cf: Deleting consensus metadata
I20260926 16:30:53.056129 8599 ts_tablet_manager.cc:1939] T 21fc895b92974a7586d46e85d951bda2 P 9cf28505064b4b4ba7581338c203ee66: Deleting consensus metadata
I20260926 16:30:53.056813 8000 catalog_manager.cc:4841] TS 6a4cdd1c228144b1a0a7bd99cab216cf (127.6.33.2:37269): tablet 21fc895b92974a7586d46e85d951bda2 (table test-table [id=572c064ca6a948db8b9e9ab257e63c3b]) successfully deleted
I20260926 16:30:53.056814 7997 catalog_manager.cc:4841] TS 49a7b8cf178d46f28c018c0bda345304 (127.6.33.3:45795): tablet 21fc895b92974a7586d46e85d951bda2 (table test-table [id=572c064ca6a948db8b9e9ab257e63c3b]) successfully deleted
I20260926 16:30:53.057287 7999 catalog_manager.cc:4841] TS 9cf28505064b4b4ba7581338c203ee66 (127.6.33.1:38069): tablet 21fc895b92974a7586d46e85d951bda2 (table test-table [id=572c064ca6a948db8b9e9ab257e63c3b]) successfully deleted
Sep 26 16:30:53 dist-test-slave-khd3 krb5kdc[7968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440253, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 16:30:53 dist-test-slave-khd3 krb5kdc[7968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440253, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:30:53.091080 8608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:53.077216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52226 (local address 127.6.33.62:41329)
0926 16:30:53.077812 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:53.077816 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:53.077839 (+ 23us) server_negotiation.cc:407] Connection header received
0926 16:30:53.077975 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:53.077979 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:53.078040 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:53.078132 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:30:53.079459 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:53.080230 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:53.081615 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:53.082045 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:53.082146 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:53.086248 (+ 4102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:30:53.086253 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:30:53.086257 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:30:53.086292 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:30:53.088716 (+ 2424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:53.089384 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:53.089388 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:53.089390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:53.089497 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:53.089787 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:53.089790 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:53.089792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:53.090060 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:30:53.090226 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:53.090647 (+ 421us) server_negotiation.cc:299] Negotiation successful
0926 16:30:53.090822 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":285,"threads_started":1}
I20260926 16:30:53.094880 8012 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52226:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:30:53.095505 8012 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:30:53.102334 8150 tablet_service.cc:1467] Processing CreateTablet for tablet a1c8755f3b3b447991ba270f8430b301 (DEFAULT_TABLE table=test-table [id=80da6b35d4d5460c97e1d20ca0f5fc8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:30:53.102315 8289 tablet_service.cc:1467] Processing CreateTablet for tablet a1c8755f3b3b447991ba270f8430b301 (DEFAULT_TABLE table=test-table [id=80da6b35d4d5460c97e1d20ca0f5fc8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:30:53.102610 8428 tablet_service.cc:1467] Processing CreateTablet for tablet a1c8755f3b3b447991ba270f8430b301 (DEFAULT_TABLE table=test-table [id=80da6b35d4d5460c97e1d20ca0f5fc8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:30:53.102813 8289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1c8755f3b3b447991ba270f8430b301. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:30:53.102837 8150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1c8755f3b3b447991ba270f8430b301. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:30:53.103005 8428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1c8755f3b3b447991ba270f8430b301. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:30:53.108712 8609 tablet_bootstrap.cc:492] T a1c8755f3b3b447991ba270f8430b301 P 9cf28505064b4b4ba7581338c203ee66: Bootstrap starting.
I20260926 16:30:53.110112 8609 tablet_bootstrap.cc:654] T a1c8755f3b3b447991ba270f8430b301 P 9cf28505064b4b4ba7581338c203ee66: Neither blocks nor log segments found. Creating new log.
I20260926 16:30:53.110924 8611 tablet_bootstrap.cc:492] T a1c8755f3b3b447991ba270f8430b301 P 49a7b8cf178d46f28c018c0bda345304: Bootstrap starting.
I20260926 16:30:53.111617 8610 tablet_bootstrap.cc:492] T a1c8755f3b3b447991ba270f8430b301 P 6a4cdd1c228144b1a0a7bd99cab216cf: Bootstrap starting.
I20260926 16:30:53.112831 8609 tablet_bootstrap.cc:492] T a1c8755f3b3b447991ba270f8430b301 P 9cf28505064b4b4ba7581338c203ee66: No bootstrap required, opened a new log
I20260926 16:30:53.113052 8609 ts_tablet_manager.cc:1397] T a1c8755f3b3b447991ba270f8430b301 P 9cf28505064b4b4ba7581338c203ee66: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260926 16:30:53.113379 8611 tablet_bootstrap.cc:654] T a1c8755f3b3b447991ba270f8430b301 P 49a7b8cf178d46f28c018c0bda345304: Neither blocks nor log segments found. Creating new log.
I20260926 16:30:53.113925 8609 raft_consensus.cc:348] T a1c8755f3b3b447991ba270f8430b301 P 9cf28505064b4b4ba7581338c203ee66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf28505064b4b4ba7581338c203ee66" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38069 } } peers { permanent_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37269 } } peers { permanent_uuid: "49a7b8cf178d46f28c018c0bda345304" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45795 } }
I20260926 16:30:53.114264 8609 raft_consensus.cc:374] T a1c8755f3b3b447991ba270f8430b301 P 9cf28505064b4b4ba7581338c203ee66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:30:53.114377 8610 tablet_bootstrap.cc:654] T a1c8755f3b3b447991ba270f8430b301 P 6a4cdd1c228144b1a0a7bd99cab216cf: Neither blocks nor log segments found. Creating new log.
I20260926 16:30:53.114395 8609 raft_consensus.cc:729] T a1c8755f3b3b447991ba270f8430b301 P 9cf28505064b4b4ba7581338c203ee66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cf28505064b4b4ba7581338c203ee66, State: Initialized, Role: FOLLOWER
I20260926 16:30:53.114710 8609 consensus_queue.cc:260] T a1c8755f3b3b447991ba270f8430b301 P 9cf28505064b4b4ba7581338c203ee66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf28505064b4b4ba7581338c203ee66" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38069 } } peers { permanent_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37269 } } peers { permanent_uuid: "49a7b8cf178d46f28c018c0bda345304" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45795 } }
I20260926 16:30:53.122892 8609 ts_tablet_manager.cc:1428] T a1c8755f3b3b447991ba270f8430b301 P 9cf28505064b4b4ba7581338c203ee66: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20260926 16:30:53.123595 8610 tablet_bootstrap.cc:492] T a1c8755f3b3b447991ba270f8430b301 P 6a4cdd1c228144b1a0a7bd99cab216cf: No bootstrap required, opened a new log
I20260926 16:30:53.123878 8610 ts_tablet_manager.cc:1397] T a1c8755f3b3b447991ba270f8430b301 P 6a4cdd1c228144b1a0a7bd99cab216cf: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.004s
I20260926 16:30:53.124711 8611 tablet_bootstrap.cc:492] T a1c8755f3b3b447991ba270f8430b301 P 49a7b8cf178d46f28c018c0bda345304: No bootstrap required, opened a new log
I20260926 16:30:53.124913 8611 ts_tablet_manager.cc:1397] T a1c8755f3b3b447991ba270f8430b301 P 49a7b8cf178d46f28c018c0bda345304: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20260926 16:30:53.124733 8610 raft_consensus.cc:348] T a1c8755f3b3b447991ba270f8430b301 P 6a4cdd1c228144b1a0a7bd99cab216cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf28505064b4b4ba7581338c203ee66" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38069 } } peers { permanent_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37269 } } peers { permanent_uuid: "49a7b8cf178d46f28c018c0bda345304" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45795 } }
I20260926 16:30:53.125000 8610 raft_consensus.cc:374] T a1c8755f3b3b447991ba270f8430b301 P 6a4cdd1c228144b1a0a7bd99cab216cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:30:53.125150 8610 raft_consensus.cc:729] T a1c8755f3b3b447991ba270f8430b301 P 6a4cdd1c228144b1a0a7bd99cab216cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a4cdd1c228144b1a0a7bd99cab216cf, State: Initialized, Role: FOLLOWER
I20260926 16:30:53.125417 8610 consensus_queue.cc:260] T a1c8755f3b3b447991ba270f8430b301 P 6a4cdd1c228144b1a0a7bd99cab216cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf28505064b4b4ba7581338c203ee66" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38069 } } peers { permanent_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37269 } } peers { permanent_uuid: "49a7b8cf178d46f28c018c0bda345304" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45795 } }
I20260926 16:30:53.125643 8611 raft_consensus.cc:348] T a1c8755f3b3b447991ba270f8430b301 P 49a7b8cf178d46f28c018c0bda345304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf28505064b4b4ba7581338c203ee66" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38069 } } peers { permanent_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37269 } } peers { permanent_uuid: "49a7b8cf178d46f28c018c0bda345304" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45795 } }
I20260926 16:30:53.125910 8611 raft_consensus.cc:374] T a1c8755f3b3b447991ba270f8430b301 P 49a7b8cf178d46f28c018c0bda345304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:30:53.126034 8611 raft_consensus.cc:729] T a1c8755f3b3b447991ba270f8430b301 P 49a7b8cf178d46f28c018c0bda345304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49a7b8cf178d46f28c018c0bda345304, State: Initialized, Role: FOLLOWER
I20260926 16:30:53.126304 8611 consensus_queue.cc:260] T a1c8755f3b3b447991ba270f8430b301 P 49a7b8cf178d46f28c018c0bda345304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf28505064b4b4ba7581338c203ee66" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38069 } } peers { permanent_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37269 } } peers { permanent_uuid: "49a7b8cf178d46f28c018c0bda345304" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45795 } }
I20260926 16:30:53.126638 8610 ts_tablet_manager.cc:1428] T a1c8755f3b3b447991ba270f8430b301 P 6a4cdd1c228144b1a0a7bd99cab216cf: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260926 16:30:53.130079 8611 ts_tablet_manager.cc:1428] T a1c8755f3b3b447991ba270f8430b301 P 49a7b8cf178d46f28c018c0bda345304: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20260926 16:30:53.382248 8616 raft_consensus.cc:482] T a1c8755f3b3b447991ba270f8430b301 P 49a7b8cf178d46f28c018c0bda345304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:30:53.382493 8616 raft_consensus.cc:504] T a1c8755f3b3b447991ba270f8430b301 P 49a7b8cf178d46f28c018c0bda345304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf28505064b4b4ba7581338c203ee66" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38069 } } peers { permanent_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37269 } } peers { permanent_uuid: "49a7b8cf178d46f28c018c0bda345304" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45795 } }
I20260926 16:30:53.384160 8616 leader_election.cc:288] T a1c8755f3b3b447991ba270f8430b301 P 49a7b8cf178d46f28c018c0bda345304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9cf28505064b4b4ba7581338c203ee66 (127.6.33.1:38069), 6a4cdd1c228144b1a0a7bd99cab216cf (127.6.33.2:37269)
I20260926 16:30:53.404121 8620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:53.384622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.3:45849 (local address 127.6.33.2:37269)
0926 16:30:53.397120 (+ 12498us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:53.397124 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:53.397145 (+ 21us) server_negotiation.cc:407] Connection header received
0926 16:30:53.397200 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:53.397230 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:53.397307 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:53.397413 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:30:53.398133 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:53.398983 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:53.401805 (+ 2822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:53.403014 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:53.403140 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:53.403256 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:53.403656 (+ 400us) server_negotiation.cc:299] Negotiation successful
0926 16:30:53.403805 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12373,"thread_start_us":12244,"threads_started":1}
I20260926 16:30:53.404142 8617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:53.384509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.3:45663 (local address 127.6.33.1:38069)
0926 16:30:53.396936 (+ 12427us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:53.396942 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:53.396969 (+ 27us) server_negotiation.cc:407] Connection header received
0926 16:30:53.397045 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:53.397048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:53.397133 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:53.397234 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:30:53.397958 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:53.398879 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:53.401959 (+ 3080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:53.403157 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:53.403284 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:53.403393 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:53.403656 (+ 263us) server_negotiation.cc:299] Negotiation successful
0926 16:30:53.403795 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12277,"thread_start_us":12078,"threads_started":1}
I20260926 16:30:53.404322 8618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:53.384518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.2:37269 (local address 127.6.33.3:45849)
0926 16:30:53.396452 (+ 11934us) negotiation.cc:107] Waiting for socket to connect
0926 16:30:53.396471 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 16:30:53.396581 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:30:53.397459 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:30:53.397462 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:30:53.397548 (+ 86us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:30:53.397930 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:53.397941 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:53.399102 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:53.399111 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:53.401619 (+ 2508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:53.401627 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:53.403193 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:53.403197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:53.403434 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:53.403501 (+ 67us) client_negotiation.cc:769] Sending connection context
0926 16:30:53.403596 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 16:30:53.403756 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11805,"mutex_wait_us":121,"spinlock_wait_cycles":7296,"thread_start_us":11670,"threads_started":1}
I20260926 16:30:53.404551 8619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:53.384497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:38069 (local address 127.6.33.3:45663)
0926 16:30:53.396137 (+ 11640us) negotiation.cc:107] Waiting for socket to connect
0926 16:30:53.396190 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 16:30:53.396397 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:30:53.397302 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:30:53.397307 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:30:53.397337 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:30:53.397776 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:53.397788 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:53.399008 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:53.399016 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:53.401612 (+ 2596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:53.401627 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:53.403338 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:53.403343 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:53.403513 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:53.403542 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 16:30:53.403618 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 16:30:53.403756 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11462,"mutex_wait_us":185,"spinlock_wait_cycles":36608,"thread_start_us":11281,"threads_started":1}
I20260926 16:30:53.405109 8309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1c8755f3b3b447991ba270f8430b301" candidate_uuid: "49a7b8cf178d46f28c018c0bda345304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" is_pre_election: true
I20260926 16:30:53.405145 8168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1c8755f3b3b447991ba270f8430b301" candidate_uuid: "49a7b8cf178d46f28c018c0bda345304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cf28505064b4b4ba7581338c203ee66" is_pre_election: true
I20260926 16:30:53.405387 8309 raft_consensus.cc:2454] T a1c8755f3b3b447991ba270f8430b301 P 6a4cdd1c228144b1a0a7bd99cab216cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49a7b8cf178d46f28c018c0bda345304 in term 0.
I20260926 16:30:53.405397 8168 raft_consensus.cc:2454] T a1c8755f3b3b447991ba270f8430b301 P 9cf28505064b4b4ba7581338c203ee66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49a7b8cf178d46f28c018c0bda345304 in term 0.
I20260926 16:30:53.405871 8382 leader_election.cc:302] T a1c8755f3b3b447991ba270f8430b301 P 49a7b8cf178d46f28c018c0bda345304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49a7b8cf178d46f28c018c0bda345304, 6a4cdd1c228144b1a0a7bd99cab216cf; no voters:
I20260926 16:30:53.406271 8616 raft_consensus.cc:2781] T a1c8755f3b3b447991ba270f8430b301 P 49a7b8cf178d46f28c018c0bda345304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:30:53.406397 8616 raft_consensus.cc:482] T a1c8755f3b3b447991ba270f8430b301 P 49a7b8cf178d46f28c018c0bda345304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:30:53.406488 8616 raft_consensus.cc:3037] T a1c8755f3b3b447991ba270f8430b301 P 49a7b8cf178d46f28c018c0bda345304 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:30:53.408752 8616 raft_consensus.cc:504] T a1c8755f3b3b447991ba270f8430b301 P 49a7b8cf178d46f28c018c0bda345304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf28505064b4b4ba7581338c203ee66" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38069 } } peers { permanent_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37269 } } peers { permanent_uuid: "49a7b8cf178d46f28c018c0bda345304" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45795 } }
I20260926 16:30:53.409446 8616 leader_election.cc:288] T a1c8755f3b3b447991ba270f8430b301 P 49a7b8cf178d46f28c018c0bda345304 [CANDIDATE]: Term 1 election: Requested vote from peers 9cf28505064b4b4ba7581338c203ee66 (127.6.33.1:38069), 6a4cdd1c228144b1a0a7bd99cab216cf (127.6.33.2:37269)
I20260926 16:30:53.409839 8168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1c8755f3b3b447991ba270f8430b301" candidate_uuid: "49a7b8cf178d46f28c018c0bda345304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cf28505064b4b4ba7581338c203ee66"
I20260926 16:30:53.409931 8309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1c8755f3b3b447991ba270f8430b301" candidate_uuid: "49a7b8cf178d46f28c018c0bda345304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf"
I20260926 16:30:53.410063 8168 raft_consensus.cc:3037] T a1c8755f3b3b447991ba270f8430b301 P 9cf28505064b4b4ba7581338c203ee66 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:30:53.410187 8309 raft_consensus.cc:3037] T a1c8755f3b3b447991ba270f8430b301 P 6a4cdd1c228144b1a0a7bd99cab216cf [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:30:53.412249 8168 raft_consensus.cc:2454] T a1c8755f3b3b447991ba270f8430b301 P 9cf28505064b4b4ba7581338c203ee66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49a7b8cf178d46f28c018c0bda345304 in term 1.
I20260926 16:30:53.412272 8309 raft_consensus.cc:2454] T a1c8755f3b3b447991ba270f8430b301 P 6a4cdd1c228144b1a0a7bd99cab216cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49a7b8cf178d46f28c018c0bda345304 in term 1.
I20260926 16:30:53.412664 8381 leader_election.cc:302] T a1c8755f3b3b447991ba270f8430b301 P 49a7b8cf178d46f28c018c0bda345304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49a7b8cf178d46f28c018c0bda345304, 9cf28505064b4b4ba7581338c203ee66; no voters:
I20260926 16:30:53.412993 8616 raft_consensus.cc:2781] T a1c8755f3b3b447991ba270f8430b301 P 49a7b8cf178d46f28c018c0bda345304 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:30:53.413523 8616 raft_consensus.cc:686] T a1c8755f3b3b447991ba270f8430b301 P 49a7b8cf178d46f28c018c0bda345304 [term 1 LEADER]: Becoming Leader. State: Replica: 49a7b8cf178d46f28c018c0bda345304, State: Running, Role: LEADER
I20260926 16:30:53.413915 8616 consensus_queue.cc:237] T a1c8755f3b3b447991ba270f8430b301 P 49a7b8cf178d46f28c018c0bda345304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf28505064b4b4ba7581338c203ee66" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38069 } } peers { permanent_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37269 } } peers { permanent_uuid: "49a7b8cf178d46f28c018c0bda345304" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45795 } }
I20260926 16:30:53.416992 8012 catalog_manager.cc:5495] T a1c8755f3b3b447991ba270f8430b301 P 49a7b8cf178d46f28c018c0bda345304 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49a7b8cf178d46f28c018c0bda345304 (127.6.33.3). New cstate: current_term: 1 leader_uuid: "49a7b8cf178d46f28c018c0bda345304" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf28505064b4b4ba7581338c203ee66" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49a7b8cf178d46f28c018c0bda345304" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45795 } health_report { overall_health: HEALTHY } } }
I20260926 16:30:53.489382 8624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:53.483293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34324 (local address 127.6.33.3:45795)
0926 16:30:53.483966 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:53.483977 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:53.483994 (+ 17us) server_negotiation.cc:407] Connection header received
0926 16:30:53.484067 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:53.484070 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:53.484141 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:53.484307 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:30:53.484746 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:53.485624 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:53.487089 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:53.487521 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:53.487676 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:53.488118 (+ 442us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:30:53.488701 (+ 583us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:30:53.488909 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:53.488986 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 16:30:53.489063 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":402,"threads_started":1}
I20260926 16:30:53.505043 8620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:53.499313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.3:52527 (local address 127.6.33.2:37269)
0926 16:30:53.499545 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:53.499553 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:53.499912 (+ 359us) server_negotiation.cc:407] Connection header received
0926 16:30:53.500073 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:53.500077 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:53.500140 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:53.500213 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:30:53.500724 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:53.501499 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:53.503402 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:53.503838 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:53.503931 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:53.504413 (+ 482us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:30:53.504512 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:30:53.504577 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:53.504706 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 16:30:53.504848 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 16:30:53.505206 8627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:53.499125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.2:37269 (local address 127.6.33.3:52527)
0926 16:30:53.499809 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0926 16:30:53.499828 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 16:30:53.499954 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:30:53.500245 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:30:53.500249 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:30:53.500261 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 16:30:53.500582 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:53.500592 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:53.501619 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:53.501625 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:53.503264 (+ 1639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:53.503278 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:53.504001 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:53.504007 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:53.504194 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:53.504257 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 16:30:53.504597 (+ 340us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 16:30:53.504607 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 16:30:53.504681 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 16:30:53.504853 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":370,"threads_started":1}
I20260926 16:30:53.513224 8168 raft_consensus.cc:1261] T a1c8755f3b3b447991ba270f8430b301 P 9cf28505064b4b4ba7581338c203ee66 [term 1 FOLLOWER]: Refusing update from remote peer 49a7b8cf178d46f28c018c0bda345304: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:30:53.513230 8309 raft_consensus.cc:1261] T a1c8755f3b3b447991ba270f8430b301 P 6a4cdd1c228144b1a0a7bd99cab216cf [term 1 FOLLOWER]: Refusing update from remote peer 49a7b8cf178d46f28c018c0bda345304: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:30:53.513926 8621 consensus_queue.cc:1035] T a1c8755f3b3b447991ba270f8430b301 P 49a7b8cf178d46f28c018c0bda345304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf28505064b4b4ba7581338c203ee66" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:30:53.514256 8616 consensus_queue.cc:1035] T a1c8755f3b3b447991ba270f8430b301 P 49a7b8cf178d46f28c018c0bda345304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:30:53.546536 8624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:53.535514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:39915 (local address 127.6.33.3:45795)
0926 16:30:53.536096 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:53.536102 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:53.541371 (+ 5269us) server_negotiation.cc:407] Connection header received
0926 16:30:53.541434 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:53.541438 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:53.541498 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:53.541578 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:30:53.541937 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:53.542808 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:53.544633 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:53.545194 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:53.545358 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:53.545775 (+ 417us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:30:53.545901 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:30:53.545981 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:53.546141 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 16:30:53.546216 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 16:30:53.546587 8632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:53.535013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.3:45795 (local address 127.6.33.2:39915)
0926 16:30:53.541184 (+ 6171us) negotiation.cc:107] Waiting for socket to connect
0926 16:30:53.541200 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:30:53.541313 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:30:53.541584 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:30:53.541587 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:30:53.541594 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 16:30:53.541794 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:53.541809 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:53.542941 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:53.542947 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:53.544460 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:53.544476 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:53.545396 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:53.545403 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:53.545546 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:53.545580 (+ 34us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 16:30:53.546006 (+ 426us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 16:30:53.546018 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 16:30:53.546116 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 16:30:53.546249 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6044,"thread_start_us":5920,"threads_started":1}
I20260926 16:30:53.577880 8011 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52226:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:30:53.578128 8011 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52226:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:30:53.581168 8011 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4ed326500513428ebd1beeff75c87965: Sending DeleteTablet for 3 replicas of tablet a1c8755f3b3b447991ba270f8430b301
I20260926 16:30:53.582261 8150 tablet_service.cc:1514] Processing DeleteTablet for tablet a1c8755f3b3b447991ba270f8430b301 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:30:53 UTC) from {username='kudu', principal='kudu/127.6.33.62@KRBTEST.COM'} at 127.0.0.1:50020
I20260926 16:30:53.582306 8289 tablet_service.cc:1514] Processing DeleteTablet for tablet a1c8755f3b3b447991ba270f8430b301 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:30:53 UTC) from {username='kudu', principal='kudu/127.6.33.62@KRBTEST.COM'} at 127.0.0.1:51128
I20260926 16:30:53.582671 8428 tablet_service.cc:1514] Processing DeleteTablet for tablet a1c8755f3b3b447991ba270f8430b301 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:30:53 UTC) from {username='kudu', principal='kudu/127.6.33.62@KRBTEST.COM'} at 127.0.0.1:34310
I20260926 16:30:53.582860 8637 tablet_replica.cc:330] T a1c8755f3b3b447991ba270f8430b301 P 6a4cdd1c228144b1a0a7bd99cab216cf: stopping tablet replica
I20260926 16:30:53.583060 8637 raft_consensus.cc:2229] T a1c8755f3b3b447991ba270f8430b301 P 6a4cdd1c228144b1a0a7bd99cab216cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:30:53.583540 8636 tablet_replica.cc:330] T a1c8755f3b3b447991ba270f8430b301 P 9cf28505064b4b4ba7581338c203ee66: stopping tablet replica
I20260926 16:30:53.583734 8637 raft_consensus.cc:2258] T a1c8755f3b3b447991ba270f8430b301 P 6a4cdd1c228144b1a0a7bd99cab216cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:30:53.583751 8636 raft_consensus.cc:2229] T a1c8755f3b3b447991ba270f8430b301 P 9cf28505064b4b4ba7581338c203ee66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:30:53.583945 8636 raft_consensus.cc:2258] T a1c8755f3b3b447991ba270f8430b301 P 9cf28505064b4b4ba7581338c203ee66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:30:53.583760 8011 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52226:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:30:53.584411 8011 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:30:53.585180 8636 ts_tablet_manager.cc:1905] T a1c8755f3b3b447991ba270f8430b301 P 9cf28505064b4b4ba7581338c203ee66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:30:53.585292 8638 tablet_replica.cc:330] T a1c8755f3b3b447991ba270f8430b301 P 49a7b8cf178d46f28c018c0bda345304: stopping tablet replica
I20260926 16:30:53.585534 8638 raft_consensus.cc:2229] T a1c8755f3b3b447991ba270f8430b301 P 49a7b8cf178d46f28c018c0bda345304 [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:30:53.586306 8638 raft_consensus.cc:2258] T a1c8755f3b3b447991ba270f8430b301 P 49a7b8cf178d46f28c018c0bda345304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:30:53.587332 8638 ts_tablet_manager.cc:1905] T a1c8755f3b3b447991ba270f8430b301 P 49a7b8cf178d46f28c018c0bda345304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:30:53.587386 8637 ts_tablet_manager.cc:1905] T a1c8755f3b3b447991ba270f8430b301 P 6a4cdd1c228144b1a0a7bd99cab216cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:30:53.594172 8428 tablet_service.cc:1467] Processing CreateTablet for tablet 5c9a6b8152b54d56971f12ca84692609 (DEFAULT_TABLE table=test-table [id=d44201941c554adba3c24cd0a548aeb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:30:53.594306 8638 ts_tablet_manager.cc:1918] T a1c8755f3b3b447991ba270f8430b301 P 49a7b8cf178d46f28c018c0bda345304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 16:30:53.594440 8638 log.cc:1196] T a1c8755f3b3b447991ba270f8430b301 P 49a7b8cf178d46f28c018c0bda345304: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/ts-2/wal/wals/a1c8755f3b3b447991ba270f8430b301
I20260926 16:30:53.594523 8150 tablet_service.cc:1467] Processing CreateTablet for tablet 5c9a6b8152b54d56971f12ca84692609 (DEFAULT_TABLE table=test-table [id=d44201941c554adba3c24cd0a548aeb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:30:53.594652 8428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c9a6b8152b54d56971f12ca84692609. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:30:53.594997 8638 ts_tablet_manager.cc:1939] T a1c8755f3b3b447991ba270f8430b301 P 49a7b8cf178d46f28c018c0bda345304: Deleting consensus metadata
I20260926 16:30:53.595036 8150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c9a6b8152b54d56971f12ca84692609. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:30:53.599964 8289 tablet_service.cc:1467] Processing CreateTablet for tablet 5c9a6b8152b54d56971f12ca84692609 (DEFAULT_TABLE table=test-table [id=d44201941c554adba3c24cd0a548aeb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:30:53.600306 8289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c9a6b8152b54d56971f12ca84692609. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:30:53.601462 7997 catalog_manager.cc:4841] TS 49a7b8cf178d46f28c018c0bda345304 (127.6.33.3:45795): tablet a1c8755f3b3b447991ba270f8430b301 (table test-table [id=80da6b35d4d5460c97e1d20ca0f5fc8a]) successfully deleted
I20260926 16:30:53.602166 8611 tablet_bootstrap.cc:492] T 5c9a6b8152b54d56971f12ca84692609 P 49a7b8cf178d46f28c018c0bda345304: Bootstrap starting.
I20260926 16:30:53.603850 8611 tablet_bootstrap.cc:654] T 5c9a6b8152b54d56971f12ca84692609 P 49a7b8cf178d46f28c018c0bda345304: Neither blocks nor log segments found. Creating new log.
I20260926 16:30:53.605312 8609 tablet_bootstrap.cc:492] T 5c9a6b8152b54d56971f12ca84692609 P 9cf28505064b4b4ba7581338c203ee66: Bootstrap starting.
I20260926 16:30:53.606760 8636 ts_tablet_manager.cc:1918] T a1c8755f3b3b447991ba270f8430b301 P 9cf28505064b4b4ba7581338c203ee66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 16:30:53.606900 8636 log.cc:1196] T a1c8755f3b3b447991ba270f8430b301 P 9cf28505064b4b4ba7581338c203ee66: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/ts-0/wal/wals/a1c8755f3b3b447991ba270f8430b301
I20260926 16:30:53.607023 8609 tablet_bootstrap.cc:654] T 5c9a6b8152b54d56971f12ca84692609 P 9cf28505064b4b4ba7581338c203ee66: Neither blocks nor log segments found. Creating new log.
I20260926 16:30:53.607467 8636 ts_tablet_manager.cc:1939] T a1c8755f3b3b447991ba270f8430b301 P 9cf28505064b4b4ba7581338c203ee66: Deleting consensus metadata
I20260926 16:30:53.608839 7999 catalog_manager.cc:4841] TS 9cf28505064b4b4ba7581338c203ee66 (127.6.33.1:38069): tablet a1c8755f3b3b447991ba270f8430b301 (table test-table [id=80da6b35d4d5460c97e1d20ca0f5fc8a]) successfully deleted
I20260926 16:30:53.609781 8637 ts_tablet_manager.cc:1918] T a1c8755f3b3b447991ba270f8430b301 P 6a4cdd1c228144b1a0a7bd99cab216cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 16:30:53.609922 8637 log.cc:1196] T a1c8755f3b3b447991ba270f8430b301 P 6a4cdd1c228144b1a0a7bd99cab216cf: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/ts-1/wal/wals/a1c8755f3b3b447991ba270f8430b301
I20260926 16:30:53.610332 8610 tablet_bootstrap.cc:492] T 5c9a6b8152b54d56971f12ca84692609 P 6a4cdd1c228144b1a0a7bd99cab216cf: Bootstrap starting.
I20260926 16:30:53.610421 8637 ts_tablet_manager.cc:1939] T a1c8755f3b3b447991ba270f8430b301 P 6a4cdd1c228144b1a0a7bd99cab216cf: Deleting consensus metadata
I20260926 16:30:53.611750 8000 catalog_manager.cc:4841] TS 6a4cdd1c228144b1a0a7bd99cab216cf (127.6.33.2:37269): tablet a1c8755f3b3b447991ba270f8430b301 (table test-table [id=80da6b35d4d5460c97e1d20ca0f5fc8a]) successfully deleted
I20260926 16:30:53.612332 8610 tablet_bootstrap.cc:654] T 5c9a6b8152b54d56971f12ca84692609 P 6a4cdd1c228144b1a0a7bd99cab216cf: Neither blocks nor log segments found. Creating new log.
I20260926 16:30:53.616938 8609 tablet_bootstrap.cc:492] T 5c9a6b8152b54d56971f12ca84692609 P 9cf28505064b4b4ba7581338c203ee66: No bootstrap required, opened a new log
I20260926 16:30:53.617240 8611 tablet_bootstrap.cc:492] T 5c9a6b8152b54d56971f12ca84692609 P 49a7b8cf178d46f28c018c0bda345304: No bootstrap required, opened a new log
I20260926 16:30:53.617260 8609 ts_tablet_manager.cc:1397] T 5c9a6b8152b54d56971f12ca84692609 P 9cf28505064b4b4ba7581338c203ee66: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20260926 16:30:53.617537 8611 ts_tablet_manager.cc:1397] T 5c9a6b8152b54d56971f12ca84692609 P 49a7b8cf178d46f28c018c0bda345304: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.008s
I20260926 16:30:53.618315 8611 raft_consensus.cc:348] T 5c9a6b8152b54d56971f12ca84692609 P 49a7b8cf178d46f28c018c0bda345304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf28505064b4b4ba7581338c203ee66" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38069 } } peers { permanent_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37269 } } peers { permanent_uuid: "49a7b8cf178d46f28c018c0bda345304" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45795 } }
I20260926 16:30:53.618614 8611 raft_consensus.cc:374] T 5c9a6b8152b54d56971f12ca84692609 P 49a7b8cf178d46f28c018c0bda345304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:30:53.618692 8611 raft_consensus.cc:729] T 5c9a6b8152b54d56971f12ca84692609 P 49a7b8cf178d46f28c018c0bda345304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49a7b8cf178d46f28c018c0bda345304, State: Initialized, Role: FOLLOWER
I20260926 16:30:53.618973 8611 consensus_queue.cc:260] T 5c9a6b8152b54d56971f12ca84692609 P 49a7b8cf178d46f28c018c0bda345304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf28505064b4b4ba7581338c203ee66" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38069 } } peers { permanent_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37269 } } peers { permanent_uuid: "49a7b8cf178d46f28c018c0bda345304" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45795 } }
I20260926 16:30:53.619637 8611 ts_tablet_manager.cc:1428] T 5c9a6b8152b54d56971f12ca84692609 P 49a7b8cf178d46f28c018c0bda345304: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 16:30:53.620265 8609 raft_consensus.cc:348] T 5c9a6b8152b54d56971f12ca84692609 P 9cf28505064b4b4ba7581338c203ee66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf28505064b4b4ba7581338c203ee66" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38069 } } peers { permanent_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37269 } } peers { permanent_uuid: "49a7b8cf178d46f28c018c0bda345304" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45795 } }
I20260926 16:30:53.620579 8609 raft_consensus.cc:374] T 5c9a6b8152b54d56971f12ca84692609 P 9cf28505064b4b4ba7581338c203ee66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:30:53.620699 8609 raft_consensus.cc:729] T 5c9a6b8152b54d56971f12ca84692609 P 9cf28505064b4b4ba7581338c203ee66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cf28505064b4b4ba7581338c203ee66, State: Initialized, Role: FOLLOWER
I20260926 16:30:53.621111 8610 tablet_bootstrap.cc:492] T 5c9a6b8152b54d56971f12ca84692609 P 6a4cdd1c228144b1a0a7bd99cab216cf: No bootstrap required, opened a new log
I20260926 16:30:53.621053 8609 consensus_queue.cc:260] T 5c9a6b8152b54d56971f12ca84692609 P 9cf28505064b4b4ba7581338c203ee66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf28505064b4b4ba7581338c203ee66" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38069 } } peers { permanent_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37269 } } peers { permanent_uuid: "49a7b8cf178d46f28c018c0bda345304" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45795 } }
I20260926 16:30:53.621325 8610 ts_tablet_manager.cc:1397] T 5c9a6b8152b54d56971f12ca84692609 P 6a4cdd1c228144b1a0a7bd99cab216cf: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.001s
I20260926 16:30:53.622561 8610 raft_consensus.cc:348] T 5c9a6b8152b54d56971f12ca84692609 P 6a4cdd1c228144b1a0a7bd99cab216cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf28505064b4b4ba7581338c203ee66" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38069 } } peers { permanent_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37269 } } peers { permanent_uuid: "49a7b8cf178d46f28c018c0bda345304" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45795 } }
I20260926 16:30:53.622848 8610 raft_consensus.cc:374] T 5c9a6b8152b54d56971f12ca84692609 P 6a4cdd1c228144b1a0a7bd99cab216cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:30:53.622941 8610 raft_consensus.cc:729] T 5c9a6b8152b54d56971f12ca84692609 P 6a4cdd1c228144b1a0a7bd99cab216cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a4cdd1c228144b1a0a7bd99cab216cf, State: Initialized, Role: FOLLOWER
I20260926 16:30:53.623237 8609 ts_tablet_manager.cc:1428] T 5c9a6b8152b54d56971f12ca84692609 P 9cf28505064b4b4ba7581338c203ee66: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260926 16:30:53.623183 8610 consensus_queue.cc:260] T 5c9a6b8152b54d56971f12ca84692609 P 6a4cdd1c228144b1a0a7bd99cab216cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf28505064b4b4ba7581338c203ee66" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38069 } } peers { permanent_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37269 } } peers { permanent_uuid: "49a7b8cf178d46f28c018c0bda345304" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45795 } }
I20260926 16:30:53.623698 8610 ts_tablet_manager.cc:1428] T 5c9a6b8152b54d56971f12ca84692609 P 6a4cdd1c228144b1a0a7bd99cab216cf: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260926 16:30:53.844726 8633 raft_consensus.cc:482] T 5c9a6b8152b54d56971f12ca84692609 P 49a7b8cf178d46f28c018c0bda345304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:30:53.845073 8633 raft_consensus.cc:504] T 5c9a6b8152b54d56971f12ca84692609 P 49a7b8cf178d46f28c018c0bda345304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf28505064b4b4ba7581338c203ee66" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38069 } } peers { permanent_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37269 } } peers { permanent_uuid: "49a7b8cf178d46f28c018c0bda345304" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45795 } }
I20260926 16:30:53.845979 8633 leader_election.cc:288] T 5c9a6b8152b54d56971f12ca84692609 P 49a7b8cf178d46f28c018c0bda345304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9cf28505064b4b4ba7581338c203ee66 (127.6.33.1:38069), 6a4cdd1c228144b1a0a7bd99cab216cf (127.6.33.2:37269)
I20260926 16:30:53.846552 8168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c9a6b8152b54d56971f12ca84692609" candidate_uuid: "49a7b8cf178d46f28c018c0bda345304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cf28505064b4b4ba7581338c203ee66" is_pre_election: true
I20260926 16:30:53.846580 8309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c9a6b8152b54d56971f12ca84692609" candidate_uuid: "49a7b8cf178d46f28c018c0bda345304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" is_pre_election: true
I20260926 16:30:53.846824 8168 raft_consensus.cc:2454] T 5c9a6b8152b54d56971f12ca84692609 P 9cf28505064b4b4ba7581338c203ee66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49a7b8cf178d46f28c018c0bda345304 in term 0.
I20260926 16:30:53.846903 8309 raft_consensus.cc:2454] T 5c9a6b8152b54d56971f12ca84692609 P 6a4cdd1c228144b1a0a7bd99cab216cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49a7b8cf178d46f28c018c0bda345304 in term 0.
I20260926 16:30:53.847313 8381 leader_election.cc:302] T 5c9a6b8152b54d56971f12ca84692609 P 49a7b8cf178d46f28c018c0bda345304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 49a7b8cf178d46f28c018c0bda345304, 6a4cdd1c228144b1a0a7bd99cab216cf, 9cf28505064b4b4ba7581338c203ee66; no voters:
I20260926 16:30:53.847582 8633 raft_consensus.cc:2781] T 5c9a6b8152b54d56971f12ca84692609 P 49a7b8cf178d46f28c018c0bda345304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:30:53.847694 8633 raft_consensus.cc:482] T 5c9a6b8152b54d56971f12ca84692609 P 49a7b8cf178d46f28c018c0bda345304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:30:53.847779 8633 raft_consensus.cc:3037] T 5c9a6b8152b54d56971f12ca84692609 P 49a7b8cf178d46f28c018c0bda345304 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:30:53.850049 8633 raft_consensus.cc:504] T 5c9a6b8152b54d56971f12ca84692609 P 49a7b8cf178d46f28c018c0bda345304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf28505064b4b4ba7581338c203ee66" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38069 } } peers { permanent_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37269 } } peers { permanent_uuid: "49a7b8cf178d46f28c018c0bda345304" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45795 } }
I20260926 16:30:53.850594 8633 leader_election.cc:288] T 5c9a6b8152b54d56971f12ca84692609 P 49a7b8cf178d46f28c018c0bda345304 [CANDIDATE]: Term 1 election: Requested vote from peers 9cf28505064b4b4ba7581338c203ee66 (127.6.33.1:38069), 6a4cdd1c228144b1a0a7bd99cab216cf (127.6.33.2:37269)
I20260926 16:30:53.851008 8168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c9a6b8152b54d56971f12ca84692609" candidate_uuid: "49a7b8cf178d46f28c018c0bda345304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cf28505064b4b4ba7581338c203ee66"
I20260926 16:30:53.851145 8309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c9a6b8152b54d56971f12ca84692609" candidate_uuid: "49a7b8cf178d46f28c018c0bda345304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf"
I20260926 16:30:53.851361 8309 raft_consensus.cc:3037] T 5c9a6b8152b54d56971f12ca84692609 P 6a4cdd1c228144b1a0a7bd99cab216cf [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:30:53.851428 8168 raft_consensus.cc:3037] T 5c9a6b8152b54d56971f12ca84692609 P 9cf28505064b4b4ba7581338c203ee66 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:30:53.854379 8168 raft_consensus.cc:2454] T 5c9a6b8152b54d56971f12ca84692609 P 9cf28505064b4b4ba7581338c203ee66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49a7b8cf178d46f28c018c0bda345304 in term 1.
I20260926 16:30:53.854748 8309 raft_consensus.cc:2454] T 5c9a6b8152b54d56971f12ca84692609 P 6a4cdd1c228144b1a0a7bd99cab216cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49a7b8cf178d46f28c018c0bda345304 in term 1.
I20260926 16:30:53.854830 8381 leader_election.cc:302] T 5c9a6b8152b54d56971f12ca84692609 P 49a7b8cf178d46f28c018c0bda345304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49a7b8cf178d46f28c018c0bda345304, 9cf28505064b4b4ba7581338c203ee66; no voters:
I20260926 16:30:53.855091 8633 raft_consensus.cc:2781] T 5c9a6b8152b54d56971f12ca84692609 P 49a7b8cf178d46f28c018c0bda345304 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:30:53.855268 8633 raft_consensus.cc:686] T 5c9a6b8152b54d56971f12ca84692609 P 49a7b8cf178d46f28c018c0bda345304 [term 1 LEADER]: Becoming Leader. State: Replica: 49a7b8cf178d46f28c018c0bda345304, State: Running, Role: LEADER
I20260926 16:30:53.855809 8633 consensus_queue.cc:237] T 5c9a6b8152b54d56971f12ca84692609 P 49a7b8cf178d46f28c018c0bda345304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf28505064b4b4ba7581338c203ee66" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38069 } } peers { permanent_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37269 } } peers { permanent_uuid: "49a7b8cf178d46f28c018c0bda345304" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45795 } }
I20260926 16:30:53.858719 8013 catalog_manager.cc:5495] T 5c9a6b8152b54d56971f12ca84692609 P 49a7b8cf178d46f28c018c0bda345304 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49a7b8cf178d46f28c018c0bda345304 (127.6.33.3). New cstate: current_term: 1 leader_uuid: "49a7b8cf178d46f28c018c0bda345304" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf28505064b4b4ba7581338c203ee66" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49a7b8cf178d46f28c018c0bda345304" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45795 } health_report { overall_health: HEALTHY } } }
I20260926 16:30:53.929572 8168 raft_consensus.cc:1261] T 5c9a6b8152b54d56971f12ca84692609 P 9cf28505064b4b4ba7581338c203ee66 [term 1 FOLLOWER]: Refusing update from remote peer 49a7b8cf178d46f28c018c0bda345304: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:30:53.930163 8309 raft_consensus.cc:1261] T 5c9a6b8152b54d56971f12ca84692609 P 6a4cdd1c228144b1a0a7bd99cab216cf [term 1 FOLLOWER]: Refusing update from remote peer 49a7b8cf178d46f28c018c0bda345304: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:30:53.930145 8633 consensus_queue.cc:1035] T 5c9a6b8152b54d56971f12ca84692609 P 49a7b8cf178d46f28c018c0bda345304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf28505064b4b4ba7581338c203ee66" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:30:53.930725 8616 consensus_queue.cc:1035] T 5c9a6b8152b54d56971f12ca84692609 P 49a7b8cf178d46f28c018c0bda345304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a4cdd1c228144b1a0a7bd99cab216cf" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:30:53.935446 8582 mvcc.cc:205] Tried to move back new op lower bound from 7333643280089829376 to 7333643279795556352. Current Snapshot: MvccSnapshot[applied={T|T < 7333643280089829376}]
W20260926 16:30:53.954457 8581 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 16:30:55.546446 8011 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52226:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:30:55.546679 8011 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52226:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:30:55.550659 8011 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4ed326500513428ebd1beeff75c87965: Sending DeleteTablet for 3 replicas of tablet 5c9a6b8152b54d56971f12ca84692609
I20260926 16:30:55.551512 8150 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c9a6b8152b54d56971f12ca84692609 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:30:55 UTC) from {username='kudu', principal='kudu/127.6.33.62@KRBTEST.COM'} at 127.0.0.1:50020
I20260926 16:30:55.551512 8289 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c9a6b8152b54d56971f12ca84692609 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:30:55 UTC) from {username='kudu', principal='kudu/127.6.33.62@KRBTEST.COM'} at 127.0.0.1:51128
I20260926 16:30:55.551625 8428 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c9a6b8152b54d56971f12ca84692609 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:30:55 UTC) from {username='kudu', principal='kudu/127.6.33.62@KRBTEST.COM'} at 127.0.0.1:34310
I20260926 16:30:55.552183 8665 tablet_replica.cc:330] T 5c9a6b8152b54d56971f12ca84692609 P 6a4cdd1c228144b1a0a7bd99cab216cf: stopping tablet replica
I20260926 16:30:55.552675 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 8084
I20260926 16:30:55.553498 8666 tablet_replica.cc:330] T 5c9a6b8152b54d56971f12ca84692609 P 9cf28505064b4b4ba7581338c203ee66: stopping tablet replica
I20260926 16:30:55.552474 8665 raft_consensus.cc:2229] T 5c9a6b8152b54d56971f12ca84692609 P 6a4cdd1c228144b1a0a7bd99cab216cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:30:55.553689 8667 tablet_replica.cc:330] T 5c9a6b8152b54d56971f12ca84692609 P 49a7b8cf178d46f28c018c0bda345304: stopping tablet replica
I20260926 16:30:55.553818 8665 raft_consensus.cc:2258] T 5c9a6b8152b54d56971f12ca84692609 P 6a4cdd1c228144b1a0a7bd99cab216cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:30:55.553841 8666 raft_consensus.cc:2229] T 5c9a6b8152b54d56971f12ca84692609 P 9cf28505064b4b4ba7581338c203ee66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:30:55.553908 8667 raft_consensus.cc:2229] T 5c9a6b8152b54d56971f12ca84692609 P 49a7b8cf178d46f28c018c0bda345304 [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:30:55.554159 8666 raft_consensus.cc:2258] T 5c9a6b8152b54d56971f12ca84692609 P 9cf28505064b4b4ba7581338c203ee66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:30:55.554545 8667 raft_consensus.cc:2258] T 5c9a6b8152b54d56971f12ca84692609 P 49a7b8cf178d46f28c018c0bda345304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:30:55.555341 8666 ts_tablet_manager.cc:1905] T 5c9a6b8152b54d56971f12ca84692609 P 9cf28505064b4b4ba7581338c203ee66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:30:55.555539 8667 ts_tablet_manager.cc:1905] T 5c9a6b8152b54d56971f12ca84692609 P 49a7b8cf178d46f28c018c0bda345304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:30:55.561609 8665 ts_tablet_manager.cc:1905] T 5c9a6b8152b54d56971f12ca84692609 P 6a4cdd1c228144b1a0a7bd99cab216cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:30:55.562386 8666 ts_tablet_manager.cc:1918] T 5c9a6b8152b54d56971f12ca84692609 P 9cf28505064b4b4ba7581338c203ee66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 16:30:55.562381 8667 ts_tablet_manager.cc:1918] T 5c9a6b8152b54d56971f12ca84692609 P 49a7b8cf178d46f28c018c0bda345304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 16:30:55.562518 8667 log.cc:1196] T 5c9a6b8152b54d56971f12ca84692609 P 49a7b8cf178d46f28c018c0bda345304: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/ts-2/wal/wals/5c9a6b8152b54d56971f12ca84692609
I20260926 16:30:55.562565 8666 log.cc:1196] T 5c9a6b8152b54d56971f12ca84692609 P 9cf28505064b4b4ba7581338c203ee66: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/ts-0/wal/wals/5c9a6b8152b54d56971f12ca84692609
I20260926 16:30:55.563054 8667 ts_tablet_manager.cc:1939] T 5c9a6b8152b54d56971f12ca84692609 P 49a7b8cf178d46f28c018c0bda345304: Deleting consensus metadata
I20260926 16:30:55.563097 8666 ts_tablet_manager.cc:1939] T 5c9a6b8152b54d56971f12ca84692609 P 9cf28505064b4b4ba7581338c203ee66: Deleting consensus metadata
I20260926 16:30:55.564211 7999 catalog_manager.cc:4841] TS 9cf28505064b4b4ba7581338c203ee66 (127.6.33.1:38069): tablet 5c9a6b8152b54d56971f12ca84692609 (table test-table [id=d44201941c554adba3c24cd0a548aeb3]) successfully deleted
I20260926 16:30:55.564270 7997 catalog_manager.cc:4841] TS 49a7b8cf178d46f28c018c0bda345304 (127.6.33.3:45795): tablet 5c9a6b8152b54d56971f12ca84692609 (table test-table [id=d44201941c554adba3c24cd0a548aeb3]) successfully deleted
I20260926 16:30:55.568017 8665 ts_tablet_manager.cc:1918] T 5c9a6b8152b54d56971f12ca84692609 P 6a4cdd1c228144b1a0a7bd99cab216cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 16:30:55.568125 8665 log.cc:1196] T 5c9a6b8152b54d56971f12ca84692609 P 6a4cdd1c228144b1a0a7bd99cab216cf: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440233850748-6276-0/minicluster-data/ts-1/wal/wals/5c9a6b8152b54d56971f12ca84692609
I20260926 16:30:55.568558 8665 ts_tablet_manager.cc:1939] T 5c9a6b8152b54d56971f12ca84692609 P 6a4cdd1c228144b1a0a7bd99cab216cf: Deleting consensus metadata
I20260926 16:30:55.569617 8000 catalog_manager.cc:4841] TS 6a4cdd1c228144b1a0a7bd99cab216cf (127.6.33.2:37269): tablet 5c9a6b8152b54d56971f12ca84692609 (table test-table [id=d44201941c554adba3c24cd0a548aeb3]) successfully deleted
Sep 26 16:30:55 dist-test-slave-khd3 krb5kdc[7968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440253, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.33.1@KRBTEST.COM
I20260926 16:30:55.583410 8669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:55.554526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50034 (local address 127.6.33.1:38069)
0926 16:30:55.564920 (+ 10394us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:55.564928 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:55.565239 (+ 311us) server_negotiation.cc:407] Connection header received
0926 16:30:55.565452 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:55.565461 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:55.565521 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:55.565631 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:30:55.566871 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:55.567586 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:55.569027 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:55.569429 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:55.569569 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:55.578191 (+ 8622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:30:55.578206 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:30:55.578222 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:30:55.578266 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:30:55.581063 (+ 2797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:55.581699 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:55.581705 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:55.581708 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:55.581844 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:55.582169 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:55.582172 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:55.582174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:55.582505 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:30:55.582703 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:55.582983 (+ 280us) server_negotiation.cc:299] Negotiation successful
0926 16:30:55.583137 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10202,"thread_start_us":10080,"threads_started":1}
W20260926 16:30:55.584077 8215 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50034
W20260926 16:30:55.584797 6276 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 8084: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 16:30:55.585012 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 8084
I20260926 16:30:55.606762 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 8224
Sep 26 16:30:55 dist-test-slave-khd3 krb5kdc[7968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440253, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.33.2@KRBTEST.COM
I20260926 16:30:55.623061 8670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:55.607523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51146 (local address 127.6.33.2:37269)
0926 16:30:55.608024 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:55.608028 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:55.608044 (+ 16us) server_negotiation.cc:407] Connection header received
0926 16:30:55.608098 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:55.608101 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:55.608175 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:55.608284 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:30:55.609706 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:55.610449 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:55.611679 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:55.612078 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:55.612169 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:55.616265 (+ 4096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:30:55.616281 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:30:55.616297 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:30:55.616338 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:30:55.619602 (+ 3264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:55.620383 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:55.620391 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:55.620393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:55.620533 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:55.621025 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:55.621029 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:55.621032 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:55.621417 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:30:55.621813 (+ 396us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:55.622081 (+ 268us) server_negotiation.cc:299] Negotiation successful
0926 16:30:55.622391 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":299,"threads_started":1}
W20260926 16:30:55.624441 8354 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51146
W20260926 16:30:55.625231 6276 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 8224: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 16:30:55.625430 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 8224
I20260926 16:30:55.651415 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 8363
Sep 26 16:30:55 dist-test-slave-khd3 krb5kdc[7968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440253, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.33.3@KRBTEST.COM
I20260926 16:30:55.666913 8671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:55.652177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34340 (local address 127.6.33.3:45795)
0926 16:30:55.652732 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:55.652736 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:55.652807 (+ 71us) server_negotiation.cc:407] Connection header received
0926 16:30:55.652875 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:55.652878 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:55.652942 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:55.653035 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:30:55.654321 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:55.655068 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:55.656275 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:55.656717 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:55.656930 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:55.661172 (+ 4242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:30:55.661187 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:30:55.661203 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:30:55.661238 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:30:55.664069 (+ 2831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:55.664794 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:55.664807 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:55.664813 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:55.664974 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:55.665347 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:55.665354 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:55.665357 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:55.665719 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:30:55.666011 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:55.666319 (+ 308us) server_negotiation.cc:299] Negotiation successful
0926 16:30:55.666551 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":263,"threads_started":1}
W20260926 16:30:55.667716 8493 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34340
W20260926 16:30:55.668373 6276 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 8363: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 16:30:55.668517 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 8363
I20260926 16:30:55.689816 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 7981
I20260926 16:30:55.690974 8053 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:30:55.805394 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 7981
2026-09-26T16:30:55Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (9057 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:30:55 dist-test-slave-khd3 krb5kdc[8675](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:30:55 dist-test-slave-khd3 krb5kdc[8675](info): set up 2 sockets
Sep 26 16:30:55 dist-test-slave-khd3 krb5kdc[8675](info): commencing operation
krb5kdc: starting...
W20260926 16:30:56.965734 6276 mini_kdc.cc:121] Time spent starting KDC: real 1.116s user 0.001s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260926 16:30:57.193256 6276 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.227s user 0.001s sys 0.005s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260926 16:30:57.495327 6276 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.302s user 0.000s sys 0.005s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260926 16:30:57.712340 6276 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.217s user 0.000s sys 0.005s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/krb5kdc/test-user.keytab.
W20260926 16:30:58.424183 6276 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.712s user 0.005s sys 0.000s
Sep 26 16:30:58 dist-test-slave-khd3 krb5kdc[8675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:30:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:30:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab.
Entry for principal kudu/127.6.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab.
I20260926 16:30:58.512188 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:40075
--webserver_interface=127.6.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:37687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.33.62:40075
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:30:58.835351 8688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:30:58.835698 8688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:30:58.835873 8688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:30:58.843107 8688 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:30:58.843261 8688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:30:58.843376 8688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:30:58.843483 8688 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:30:58.843571 8688 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:30:58.853413 8688 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:37687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.33.62:40075
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:40075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:30:58.854779 8688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:30:58.856094 8688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:30:58.868260 8694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:30:58.868286 8696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:30:58.868527 8688 server_base.cc:1034] running on GCE node
W20260926 16:30:58.869016 8693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:30:58.869946 8688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:30:58.871306 8688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:30:58.872512 8688 hybrid_clock.cc:648] HybridClock initialized: now 1790440258872442 us; error 73 us; skew 500 ppm
I20260926 16:30:58.872905 8688 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:30:58 dist-test-slave-khd3 krb5kdc[8675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440258, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:30:58.876006 8688 init.cc:377] Logged in from keytab as kudu/127.6.33.62@KRBTEST.COM (short username kudu)
I20260926 16:30:58.877580 8688 webserver.cc:460] Webserver started at http://127.6.33.62:35107/ using document root <none> and password file <none>
I20260926 16:30:58.878285 8688 fs_manager.cc:359] Metadata directory not provided
I20260926 16:30:58.878412 8688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:30:58.878746 8688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:30:58.881788 8688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/minicluster-data/master-0/data/instance:
uuid: "a4f60bc0808a4566abb63e134c45800f"
format_stamp: "Formatted at 2026-09-26 16:30:58 on dist-test-slave-khd3"
server_key: "ec961dc1b231a63349f954de2b67aecd"
server_key_iv: "26543f563f05b8e95e1b508c9b0d2087"
server_key_version: "encryptionkey@0"
I20260926 16:30:58.882571 8688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/minicluster-data/master-0/wal/instance:
uuid: "a4f60bc0808a4566abb63e134c45800f"
format_stamp: "Formatted at 2026-09-26 16:30:58 on dist-test-slave-khd3"
server_key: "ec961dc1b231a63349f954de2b67aecd"
server_key_iv: "26543f563f05b8e95e1b508c9b0d2087"
server_key_version: "encryptionkey@0"
I20260926 16:30:58.887944 8688 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 16:30:58.891433 8703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:30:58.892923 8688 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:30:58.893095 8688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/minicluster-data/master-0/wal
uuid: "a4f60bc0808a4566abb63e134c45800f"
format_stamp: "Formatted at 2026-09-26 16:30:58 on dist-test-slave-khd3"
server_key: "ec961dc1b231a63349f954de2b67aecd"
server_key_iv: "26543f563f05b8e95e1b508c9b0d2087"
server_key_version: "encryptionkey@0"
I20260926 16:30:58.893307 8688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:30:58.906517 8688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:30:58.911528 8688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:30:58.911823 8688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:30:58.928922 8688 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.62:40075
I20260926 16:30:58.928956 8754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.62:40075 every 8 connection(s)
I20260926 16:30:58.930382 8688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
I20260926 16:30:58.934037 8755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:30:58.940644 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 8688
I20260926 16:30:58.940881 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/minicluster-data/master-0/wal/instance
I20260926 16:30:58.941532 6276 external_mini_cluster.cc:1428] Setting key c6bc37eb981b8c1963d37ef4014d84e7
I20260926 16:30:58.943970 8755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4f60bc0808a4566abb63e134c45800f: Bootstrap starting.
I20260926 16:30:58.947108 8755 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4f60bc0808a4566abb63e134c45800f: Neither blocks nor log segments found. Creating new log.
I20260926 16:30:58.948417 8755 log.cc:824] T 00000000000000000000000000000000 P a4f60bc0808a4566abb63e134c45800f: Log is configured to *not* fsync() on all Append() calls
I20260926 16:30:58.951481 8755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4f60bc0808a4566abb63e134c45800f: No bootstrap required, opened a new log
Sep 26 16:30:58 dist-test-slave-khd3 krb5kdc[8675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:30:58.960496 8755 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4f60bc0808a4566abb63e134c45800f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4f60bc0808a4566abb63e134c45800f" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 40075 } }
I20260926 16:30:58.960943 8755 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4f60bc0808a4566abb63e134c45800f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:30:58.961051 8755 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4f60bc0808a4566abb63e134c45800f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4f60bc0808a4566abb63e134c45800f, State: Initialized, Role: FOLLOWER
I20260926 16:30:58.961689 8755 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4f60bc0808a4566abb63e134c45800f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4f60bc0808a4566abb63e134c45800f" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 40075 } }
I20260926 16:30:58.961927 8755 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4f60bc0808a4566abb63e134c45800f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:30:58.962073 8755 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4f60bc0808a4566abb63e134c45800f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:30:58.962250 8755 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4f60bc0808a4566abb63e134c45800f [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:30:58.965231 8758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:58.943582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53500 (local address 127.6.33.62:40075)
0926 16:30:58.944349 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:58.944360 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:58.944400 (+ 40us) server_negotiation.cc:407] Connection header received
0926 16:30:58.946147 (+ 1747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:58.946170 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:58.946533 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:58.946842 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:30:58.948692 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:58.950129 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:58.951847 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:58.952518 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:58.952658 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:58.957205 (+ 4547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:30:58.957213 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:30:58.957229 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:30:58.957266 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:30:58.960490 (+ 3224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:58.961423 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:58.961439 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:58.961447 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:58.961604 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:58.961983 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:58.961990 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:58.961994 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:58.962724 (+ 730us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:30:58.963054 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:58.963436 (+ 382us) server_negotiation.cc:299] Negotiation successful
0926 16:30:58.963874 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":362,"threads_started":1}
I20260926 16:30:58.965473 8755 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4f60bc0808a4566abb63e134c45800f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4f60bc0808a4566abb63e134c45800f" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 40075 } }
I20260926 16:30:58.966025 8755 leader_election.cc:302] T 00000000000000000000000000000000 P a4f60bc0808a4566abb63e134c45800f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4f60bc0808a4566abb63e134c45800f; no voters:
I20260926 16:30:58.966689 8755 leader_election.cc:288] T 00000000000000000000000000000000 P a4f60bc0808a4566abb63e134c45800f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:30:58.966923 8760 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4f60bc0808a4566abb63e134c45800f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:30:58.968201 8760 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4f60bc0808a4566abb63e134c45800f [term 1 LEADER]: Becoming Leader. State: Replica: a4f60bc0808a4566abb63e134c45800f, State: Running, Role: LEADER
I20260926 16:30:58.968750 8760 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4f60bc0808a4566abb63e134c45800f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4f60bc0808a4566abb63e134c45800f" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 40075 } }
I20260926 16:30:58.968953 8755 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4f60bc0808a4566abb63e134c45800f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:30:58.971335 8762 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4f60bc0808a4566abb63e134c45800f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4f60bc0808a4566abb63e134c45800f. Latest consensus state: current_term: 1 leader_uuid: "a4f60bc0808a4566abb63e134c45800f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4f60bc0808a4566abb63e134c45800f" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 40075 } } }
I20260926 16:30:58.971731 8762 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4f60bc0808a4566abb63e134c45800f [sys.catalog]: This master's current role is: LEADER
I20260926 16:30:58.972254 8761 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4f60bc0808a4566abb63e134c45800f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4f60bc0808a4566abb63e134c45800f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4f60bc0808a4566abb63e134c45800f" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 40075 } } }
I20260926 16:30:58.972555 8761 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4f60bc0808a4566abb63e134c45800f [sys.catalog]: This master's current role is: LEADER
I20260926 16:30:58.977581 8767 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:30:58.983224 8767 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:30:58.991843 8767 catalog_manager.cc:1347] Generated new cluster ID: 7f1b3a23826c478d859f94332a614c68
I20260926 16:30:58.991957 8767 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:30:58.999643 8767 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:30:59.001484 8767 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:30:59.014024 8767 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4f60bc0808a4566abb63e134c45800f: Generated new TSK 0
I20260926 16:30:59.014897 8767 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.33.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab.
Entry for principal kudu/127.6.33.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab.
I20260926 16:30:59.072047 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.1:0
--local_ip_for_outbound_sockets=127.6.33.1
--webserver_interface=127.6.33.1
--webserver_port=0
--tserver_master_addrs=127.6.33.62:40075
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab
--principal=kudu/127.6.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:37687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:30:59.417424 8782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:30:59.417696 8782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:30:59.417878 8782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:30:59.425252 8782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:30:59.425384 8782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:30:59.425554 8782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.1
I20260926 16:30:59.435664 8782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:37687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab
--principal=kudu/127.6.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.33.1
--webserver_port=0
--tserver_master_addrs=127.6.33.62:40075
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.1
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:30:59.437157 8782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:30:59.438553 8782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:30:59.451506 8787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:30:59.451648 8790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:30:59.451659 8788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:30:59.452958 8782 server_base.cc:1034] running on GCE node
I20260926 16:30:59.453567 8782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:30:59.454519 8782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:30:59.455744 8782 hybrid_clock.cc:648] HybridClock initialized: now 1790440259455700 us; error 34 us; skew 500 ppm
I20260926 16:30:59.456067 8782 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:30:59 dist-test-slave-khd3 krb5kdc[8675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440259, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:30:59.459355 8782 init.cc:377] Logged in from keytab as kudu/127.6.33.1@KRBTEST.COM (short username kudu)
I20260926 16:30:59.460877 8782 webserver.cc:460] Webserver started at http://127.6.33.1:35407/ using document root <none> and password file <none>
I20260926 16:30:59.461567 8782 fs_manager.cc:359] Metadata directory not provided
I20260926 16:30:59.461709 8782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:30:59.462023 8782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:30:59.465051 8782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-0/data/instance:
uuid: "523084a9792442a881fabd641664356e"
format_stamp: "Formatted at 2026-09-26 16:30:59 on dist-test-slave-khd3"
server_key: "f648ed8d076d498fb71a700ed0f1a120"
server_key_iv: "546f2a58f25b1fa18bada4c5436d9791"
server_key_version: "encryptionkey@0"
I20260926 16:30:59.465945 8782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-0/wal/instance:
uuid: "523084a9792442a881fabd641664356e"
format_stamp: "Formatted at 2026-09-26 16:30:59 on dist-test-slave-khd3"
server_key: "f648ed8d076d498fb71a700ed0f1a120"
server_key_iv: "546f2a58f25b1fa18bada4c5436d9791"
server_key_version: "encryptionkey@0"
I20260926 16:30:59.471097 8782 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 16:30:59.474460 8797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:30:59.475801 8782 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 16:30:59.475970 8782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-0/wal
uuid: "523084a9792442a881fabd641664356e"
format_stamp: "Formatted at 2026-09-26 16:30:59 on dist-test-slave-khd3"
server_key: "f648ed8d076d498fb71a700ed0f1a120"
server_key_iv: "546f2a58f25b1fa18bada4c5436d9791"
server_key_version: "encryptionkey@0"
I20260926 16:30:59.476162 8782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:30:59.496618 8782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:30:59.501348 8782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:30:59.501641 8782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:30:59.502599 8782 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:30:59.503896 8782 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:30:59.503998 8782 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:30:59.504155 8782 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:30:59.504227 8782 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:30:59.537449 8782 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.1:44355
I20260926 16:30:59.537505 8909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.1:44355 every 8 connection(s)
I20260926 16:30:59.538823 8782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
I20260926 16:30:59.543140 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 8782
I20260926 16:30:59.543342 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-0/wal/instance
I20260926 16:30:59.544212 6276 external_mini_cluster.cc:1428] Setting key dc62c7a72d4763a59d305a24fadb8b0a
Sep 26 16:30:59 dist-test-slave-khd3 krb5kdc[8675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440259, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.1@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:30:59.562798 8912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:59.541570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:33493 (local address 127.6.33.62:40075)
0926 16:30:59.542040 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0926 16:30:59.542045 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:30:59.542999 (+ 954us) server_negotiation.cc:407] Connection header received
0926 16:30:59.545692 (+ 2693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:30:59.545699 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:30:59.545831 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:30:59.545940 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:30:59.548139 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:59.549076 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:59.551640 (+ 2564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:30:59.552165 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:30:59.552267 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:59.556889 (+ 4622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:30:59.556894 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:30:59.556897 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:30:59.556929 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:30:59.559441 (+ 2512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:59.560408 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:59.560412 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:59.560414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:59.560524 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:30:59.561080 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:30:59.561083 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:30:59.561085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:30:59.561329 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:30:59.561437 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 16:30:59.562379 (+ 942us) server_negotiation.cc:299] Negotiation successful
0926 16:30:59.562543 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":250,"threads_started":1}
WARNING: no policy specified for kudu/127.6.33.2@KRBTEST.COM; defaulting to no policy
I20260926 16:30:59.564416 8913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:30:59.541964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:40075 (local address 127.6.33.1:33493)
0926 16:30:59.542768 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0926 16:30:59.542832 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0926 16:30:59.545361 (+ 2529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:30:59.546279 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:30:59.546289 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:30:59.547060 (+ 771us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:30:59.547937 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:59.547977 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:59.549238 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:59.549246 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:59.551491 (+ 2245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:30:59.551509 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:30:59.552417 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:30:59.552426 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:30:59.552792 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:30:59.553536 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:30:59.553575 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:30:59.556577 (+ 3002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:30:59.559582 (+ 3005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:30:59.559603 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:30:59.559624 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:30:59.560289 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:30:59.560644 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:30:59.560651 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:30:59.560658 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:30:59.560970 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:30:59.561461 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:30:59.561470 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:30:59.561950 (+ 480us) client_negotiation.cc:769] Sending connection context
0926 16:30:59.562296 (+ 346us) client_negotiation.cc:240] Negotiation successful
0926 16:30:59.562878 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":287,"threads_started":1}
I20260926 16:30:59.566063 8910 heartbeater.cc:347] Connected to a master server at 127.6.33.62:40075
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.2@KRBTEST.COM" created.
I20260926 16:30:59.566409 8910 heartbeater.cc:464] Registering TS with master...
I20260926 16:30:59.567305 8910 heartbeater.cc:511] Master 127.6.33.62:40075 requested a full tablet report, sending...
I20260926 16:30:59.569677 8720 ts_manager.cc:194] Registered new tserver with Master: 523084a9792442a881fabd641664356e (127.6.33.1:44355)
I20260926 16:30:59.571342 8720 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.33.1@KRBTEST.COM'} at 127.6.33.1:33493
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab.
Entry for principal kudu/127.6.33.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab.
I20260926 16:30:59.589874 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.2:0
--local_ip_for_outbound_sockets=127.6.33.2
--webserver_interface=127.6.33.2
--webserver_port=0
--tserver_master_addrs=127.6.33.62:40075
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab
--principal=kudu/127.6.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:37687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:30:59.919721 8916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:30:59.920019 8916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:30:59.920110 8916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:30:59.927443 8916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:30:59.927560 8916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:30:59.927677 8916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.2
I20260926 16:30:59.942904 8916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:37687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab
--principal=kudu/127.6.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.33.2
--webserver_port=0
--tserver_master_addrs=127.6.33.62:40075
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.2
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:30:59.944499 8916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:30:59.946197 8916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:30:59.961355 8922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:30:59.961472 8921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:30:59.961469 8924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:30:59.963763 8916 server_base.cc:1034] running on GCE node
I20260926 16:30:59.964484 8916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:30:59.965654 8916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:30:59.966908 8916 hybrid_clock.cc:648] HybridClock initialized: now 1790440259966853 us; error 46 us; skew 500 ppm
I20260926 16:30:59.967249 8916 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:30:59 dist-test-slave-khd3 krb5kdc[8675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440259, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:30:59.970808 8916 init.cc:377] Logged in from keytab as kudu/127.6.33.2@KRBTEST.COM (short username kudu)
I20260926 16:30:59.972347 8916 webserver.cc:460] Webserver started at http://127.6.33.2:35863/ using document root <none> and password file <none>
I20260926 16:30:59.973119 8916 fs_manager.cc:359] Metadata directory not provided
I20260926 16:30:59.973223 8916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:30:59.973523 8916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:30:59.976734 8916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-1/data/instance:
uuid: "67b9428f1e064689bb65330acabf2505"
format_stamp: "Formatted at 2026-09-26 16:30:59 on dist-test-slave-khd3"
server_key: "36d7465af6df6435de240c0e9fb0f2e0"
server_key_iv: "5bbddd03b631073e133dd3ab839cca20"
server_key_version: "encryptionkey@0"
I20260926 16:30:59.977644 8916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-1/wal/instance:
uuid: "67b9428f1e064689bb65330acabf2505"
format_stamp: "Formatted at 2026-09-26 16:30:59 on dist-test-slave-khd3"
server_key: "36d7465af6df6435de240c0e9fb0f2e0"
server_key_iv: "5bbddd03b631073e133dd3ab839cca20"
server_key_version: "encryptionkey@0"
I20260926 16:30:59.983345 8916 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 16:30:59.987097 8931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:30:59.988891 8916 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 16:30:59.989063 8916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-1/wal
uuid: "67b9428f1e064689bb65330acabf2505"
format_stamp: "Formatted at 2026-09-26 16:30:59 on dist-test-slave-khd3"
server_key: "36d7465af6df6435de240c0e9fb0f2e0"
server_key_iv: "5bbddd03b631073e133dd3ab839cca20"
server_key_version: "encryptionkey@0"
I20260926 16:30:59.989269 8916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:31:00.007869 8916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:31:00.012815 8916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:31:00.013123 8916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:31:00.014158 8916 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:31:00.015550 8916 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:31:00.015625 8916 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:00.015758 8916 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:31:00.015808 8916 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:00.050333 8916 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.2:40189
I20260926 16:31:00.050355 9043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.2:40189 every 8 connection(s)
I20260926 16:31:00.051676 8916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
I20260926 16:31:00.059396 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 8916
I20260926 16:31:00.059574 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-1/wal/instance
I20260926 16:31:00.060030 6276 external_mini_cluster.cc:1428] Setting key 1cfd6c70dcf54e1ff40e2624b59ad8ca
Sep 26 16:31:00 dist-test-slave-khd3 krb5kdc[8675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440259, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.2@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:31:00.072465 8912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:00.054407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:58029 (local address 127.6.33.62:40075)
0926 16:31:00.054621 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:00.054628 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:00.055821 (+ 1193us) server_negotiation.cc:407] Connection header received
0926 16:31:00.057699 (+ 1878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:00.057704 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:00.057783 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:00.057872 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:00.059740 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:00.060489 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:00.062253 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:00.062679 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:00.062817 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:00.066247 (+ 3430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:00.066256 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:00.066264 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:00.066310 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:00.069004 (+ 2694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:00.069994 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:00.069999 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:00.070003 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:00.070131 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:00.070693 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:00.070698 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:00.070701 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:00.070974 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:00.071127 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:00.071869 (+ 742us) server_negotiation.cc:299] Negotiation successful
0926 16:31:00.072128 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 16:31:00.073622 9046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:00.054921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:40075 (local address 127.6.33.2:58029)
0926 16:31:00.055640 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:00.055680 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:00.057429 (+ 1749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:00.058102 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:00.058110 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:00.058666 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:00.059512 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:00.059541 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:00.060647 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:00.060657 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:00.062102 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:00.062114 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:00.062907 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:00.062913 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:00.063135 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:00.063816 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:00.063843 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:00.065999 (+ 2156us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:31:00.069173 (+ 3174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:00.069185 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:00.069205 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:00.069843 (+ 638us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:00.070291 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:00.070295 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:00.070297 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:00.070578 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:00.071104 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:31:00.071112 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:31:00.071465 (+ 353us) client_negotiation.cc:769] Sending connection context
0926 16:31:00.071737 (+ 272us) client_negotiation.cc:240] Negotiation successful
0926 16:31:00.072304 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":304,"threads_started":1}
I20260926 16:31:00.075238 9044 heartbeater.cc:347] Connected to a master server at 127.6.33.62:40075
I20260926 16:31:00.075542 9044 heartbeater.cc:464] Registering TS with master...
I20260926 16:31:00.076287 9044 heartbeater.cc:511] Master 127.6.33.62:40075 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.33.3@KRBTEST.COM; defaulting to no policy
I20260926 16:31:00.077841 8720 ts_manager.cc:194] Registered new tserver with Master: 67b9428f1e064689bb65330acabf2505 (127.6.33.2:40189)
I20260926 16:31:00.078852 8720 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.33.2@KRBTEST.COM'} at 127.6.33.2:58029
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab.
Entry for principal kudu/127.6.33.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab.
I20260926 16:31:00.102104 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.3:0
--local_ip_for_outbound_sockets=127.6.33.3
--webserver_interface=127.6.33.3
--webserver_port=0
--tserver_master_addrs=127.6.33.62:40075
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab
--principal=kudu/127.6.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:37687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:31:00.430689 9049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:31:00.430998 9049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:31:00.431173 9049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:31:00.438627 9049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:31:00.438750 9049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:31:00.438925 9049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.3
I20260926 16:31:00.448822 9049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:37687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab
--principal=kudu/127.6.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.33.3
--webserver_port=0
--tserver_master_addrs=127.6.33.62:40075
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.3
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:31:00.450222 9049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:31:00.451517 9049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:31:00.465138 9054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:00.465138 9055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:00.465453 9057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:31:00.466346 9049 server_base.cc:1034] running on GCE node
I20260926 16:31:00.466975 9049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:31:00.468014 9049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:31:00.469310 9049 hybrid_clock.cc:648] HybridClock initialized: now 1790440260469225 us; error 62 us; skew 500 ppm
I20260926 16:31:00.469652 9049 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:31:00 dist-test-slave-khd3 krb5kdc[8675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440260, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:31:00.472993 9049 init.cc:377] Logged in from keytab as kudu/127.6.33.3@KRBTEST.COM (short username kudu)
I20260926 16:31:00.474488 9049 webserver.cc:460] Webserver started at http://127.6.33.3:40775/ using document root <none> and password file <none>
I20260926 16:31:00.475224 9049 fs_manager.cc:359] Metadata directory not provided
I20260926 16:31:00.475373 9049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:31:00.475692 9049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:31:00.478937 9049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-2/data/instance:
uuid: "3f4e7eeb5c72483abf5546a674336722"
format_stamp: "Formatted at 2026-09-26 16:31:00 on dist-test-slave-khd3"
server_key: "80ed8721b229bb3db841a5ef3cb387ca"
server_key_iv: "052811e5ed878a3dd489bf30ae519a4e"
server_key_version: "encryptionkey@0"
I20260926 16:31:00.479836 9049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-2/wal/instance:
uuid: "3f4e7eeb5c72483abf5546a674336722"
format_stamp: "Formatted at 2026-09-26 16:31:00 on dist-test-slave-khd3"
server_key: "80ed8721b229bb3db841a5ef3cb387ca"
server_key_iv: "052811e5ed878a3dd489bf30ae519a4e"
server_key_version: "encryptionkey@0"
I20260926 16:31:00.485271 9049 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260926 16:31:00.488863 9064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:00.490238 9049 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:31:00.490420 9049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-2/wal
uuid: "3f4e7eeb5c72483abf5546a674336722"
format_stamp: "Formatted at 2026-09-26 16:31:00 on dist-test-slave-khd3"
server_key: "80ed8721b229bb3db841a5ef3cb387ca"
server_key_iv: "052811e5ed878a3dd489bf30ae519a4e"
server_key_version: "encryptionkey@0"
I20260926 16:31:00.490625 9049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:31:00.534022 9049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:31:00.538383 9049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:31:00.538754 9049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:31:00.539746 9049 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:31:00.541164 9049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:31:00.541270 9049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:00.541420 9049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:31:00.541492 9049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:00.574569 9049 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.3:34707
I20260926 16:31:00.574609 8910 heartbeater.cc:503] Master 127.6.33.62:40075 was elected leader, sending a full tablet report...
I20260926 16:31:00.574587 9176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.3:34707 every 8 connection(s)
I20260926 16:31:00.576062 9049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
I20260926 16:31:00.580928 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 9049
I20260926 16:31:00.581102 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-2/wal/instance
I20260926 16:31:00.581532 6276 external_mini_cluster.cc:1428] Setting key aac7ad0b98039117926b8fc51699ade0
Sep 26 16:31:00 dist-test-slave-khd3 krb5kdc[8675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440260, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.3@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:31:00.602707 9179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:00.578614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.3:59071 (local address 127.6.33.62:40075)
0926 16:31:00.579111 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:00.579116 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:00.580293 (+ 1177us) server_negotiation.cc:407] Connection header received
0926 16:31:00.582478 (+ 2185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:00.582484 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:00.582578 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:00.582694 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:00.584185 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:00.585044 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:00.586612 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:00.587016 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:00.587150 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:00.596497 (+ 9347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:00.596503 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:00.596506 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:00.596542 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:00.599540 (+ 2998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:00.600308 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:00.600313 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:00.600317 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:00.600443 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:00.600931 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:00.600937 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:00.600940 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:00.601219 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:00.601371 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:00.602105 (+ 734us) server_negotiation.cc:299] Negotiation successful
0926 16:31:00.602318 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":237,"threads_started":1}
I20260926 16:31:00.603706 9180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:00.579361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:40075 (local address 127.6.33.3:59071)
0926 16:31:00.580109 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:00.580149 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:00.582179 (+ 2030us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:00.582939 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:00.582947 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:00.583457 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:00.584014 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:00.584031 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:00.585191 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:00.585203 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:00.586466 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:00.586478 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:00.587240 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:00.587247 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:00.587465 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:00.593182 (+ 5717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:00.593210 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:00.596221 (+ 3011us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:31:00.599699 (+ 3478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:00.599710 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:00.599728 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:00.600177 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:00.600555 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:00.600561 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:00.600564 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:00.600814 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:00.601352 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:31:00.601359 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:31:00.601730 (+ 371us) client_negotiation.cc:769] Sending connection context
0926 16:31:00.602004 (+ 274us) client_negotiation.cc:240] Negotiation successful
0926 16:31:00.602580 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":266,"threads_started":1}
I20260926 16:31:00.605160 9177 heartbeater.cc:347] Connected to a master server at 127.6.33.62:40075
I20260926 16:31:00.605511 9177 heartbeater.cc:464] Registering TS with master...
I20260926 16:31:00.606330 9177 heartbeater.cc:511] Master 127.6.33.62:40075 requested a full tablet report, sending...
I20260926 16:31:00.607847 8720 ts_manager.cc:194] Registered new tserver with Master: 3f4e7eeb5c72483abf5546a674336722 (127.6.33.3:34707)
I20260926 16:31:00.608743 8720 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.33.3@KRBTEST.COM'} at 127.6.33.3:59071
I20260926 16:31:00.609637 6276 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 16:31:00.640049 9179 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:31:00.637557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53512 (local address 127.6.33.62:40075)
0926 16:31:00.637796 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:00.637805 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:00.638196 (+ 391us) server_negotiation.cc:407] Connection header received
0926 16:31:00.638346 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:00.638350 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:00.638416 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:00.638501 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:00.639783 (+ 1282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53512: BlockingRecv error: recv got EOF from 127.0.0.1:53512 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 16:31:00.642971 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 8782
W20260926 16:31:00.645861 6276 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 8782: Not authorized: Client connection negotiation failed: client connection to 127.6.33.1:44355: server requires authentication, but client does not have Kerberos credentials available
I20260926 16:31:00.646188 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 8782
I20260926 16:31:00.661616 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 8916
W20260926 16:31:00.664306 6276 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 8916: Not authorized: Client connection negotiation failed: client connection to 127.6.33.2:40189: server requires authentication, but client does not have Kerberos credentials available
I20260926 16:31:00.664455 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 8916
I20260926 16:31:00.679966 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 9049
W20260926 16:31:00.682780 6276 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 9049: Not authorized: Client connection negotiation failed: client connection to 127.6.33.3:34707: server requires authentication, but client does not have Kerberos credentials available
I20260926 16:31:00.682955 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 9049
I20260926 16:31:00.700105 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 8688
I20260926 16:31:00.701373 8750 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:31:00.803543 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 8688
2026-09-26T16:31:00Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (4994 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440233850748-6276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:31:00 dist-test-slave-khd3 krb5kdc[9195](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:31:00 dist-test-slave-khd3 krb5kdc[9195](info): set up 2 sockets
Sep 26 16:31:00 dist-test-slave-khd3 krb5kdc[9195](info): commencing operation
krb5kdc: starting...
W20260926 16:31:01.898190 6276 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Sep 26 16:31:02 dist-test-slave-khd3 krb5kdc[9195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:31:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:31:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab.
Entry for principal kudu/127.6.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab.
I20260926 16:31:02.116905 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440233850748-6276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440233850748-6276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440233850748-6276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:35479
--webserver_interface=127.6.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:40217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.33.62:35479
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:31:02.438851 9208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:31:02.439236 9208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:31:02.439407 9208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:31:02.447121 9208 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:31:02.447307 9208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:31:02.447432 9208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:31:02.447527 9208 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:31:02.447618 9208 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:31:02.457824 9208 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:40217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440233850748-6276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440233850748-6276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.33.62:35479
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:35479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440233850748-6276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:31:02.459239 9208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:31:02.460634 9208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:31:02.472378 9213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:02.472532 9214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:02.472409 9216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:31:02.473537 9208 server_base.cc:1034] running on GCE node
I20260926 16:31:02.474359 9208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:31:02.475824 9208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:31:02.477057 9208 hybrid_clock.cc:648] HybridClock initialized: now 1790440262477013 us; error 37 us; skew 500 ppm
I20260926 16:31:02.477434 9208 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:31:02 dist-test-slave-khd3 krb5kdc[9195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440262, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:31:02.480557 9208 init.cc:377] Logged in from keytab as kudu/127.6.33.62@KRBTEST.COM (short username kudu)
I20260926 16:31:02.482170 9208 webserver.cc:460] Webserver started at http://127.6.33.62:36513/ using document root <none> and password file <none>
I20260926 16:31:02.482875 9208 fs_manager.cc:359] Metadata directory not provided
I20260926 16:31:02.483021 9208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:31:02.483325 9208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:31:02.486315 9208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440233850748-6276-0/minicluster-data/master-0/data/instance:
uuid: "9df010c358234af3923f19c6bed4beb1"
format_stamp: "Formatted at 2026-09-26 16:31:02 on dist-test-slave-khd3"
server_key: "d823d15559baf8403aa6ac5dce58794e"
server_key_iv: "20e242d417f2c91a5165c59c64f6e10d"
server_key_version: "encryptionkey@0"
I20260926 16:31:02.487106 9208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440233850748-6276-0/minicluster-data/master-0/wal/instance:
uuid: "9df010c358234af3923f19c6bed4beb1"
format_stamp: "Formatted at 2026-09-26 16:31:02 on dist-test-slave-khd3"
server_key: "d823d15559baf8403aa6ac5dce58794e"
server_key_iv: "20e242d417f2c91a5165c59c64f6e10d"
server_key_version: "encryptionkey@0"
I20260926 16:31:02.492314 9208 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260926 16:31:02.495808 9223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:02.497280 9208 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:31:02.497464 9208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440233850748-6276-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440233850748-6276-0/minicluster-data/master-0/wal
uuid: "9df010c358234af3923f19c6bed4beb1"
format_stamp: "Formatted at 2026-09-26 16:31:02 on dist-test-slave-khd3"
server_key: "d823d15559baf8403aa6ac5dce58794e"
server_key_iv: "20e242d417f2c91a5165c59c64f6e10d"
server_key_version: "encryptionkey@0"
I20260926 16:31:02.497660 9208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440233850748-6276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440233850748-6276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440233850748-6276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:31:02.524013 9208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:31:02.528326 9208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:31:02.528640 9208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:31:02.548259 9208 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.62:35479
I20260926 16:31:02.548287 9274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.62:35479 every 8 connection(s)
I20260926 16:31:02.549726 9208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
I20260926 16:31:02.553833 9275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:31:02.556051 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 9208
I20260926 16:31:02.556298 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440233850748-6276-0/minicluster-data/master-0/wal/instance
I20260926 16:31:02.557123 6276 external_mini_cluster.cc:1428] Setting key f209fb7f7390d26a108c8677e4725364
I20260926 16:31:02.567829 9275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9df010c358234af3923f19c6bed4beb1: Bootstrap starting.
I20260926 16:31:02.572250 9275 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9df010c358234af3923f19c6bed4beb1: Neither blocks nor log segments found. Creating new log.
Sep 26 16:31:02 dist-test-slave-khd3 krb5kdc[9195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:31:02.573730 9275 log.cc:824] T 00000000000000000000000000000000 P 9df010c358234af3923f19c6bed4beb1: Log is configured to *not* fsync() on all Append() calls
I20260926 16:31:02.577512 9275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9df010c358234af3923f19c6bed4beb1: No bootstrap required, opened a new log
I20260926 16:31:02.582772 9278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:02.559210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60322 (local address 127.6.33.62:35479)
0926 16:31:02.560112 (+ 902us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:02.560128 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:02.560178 (+ 50us) server_negotiation.cc:407] Connection header received
0926 16:31:02.562029 (+ 1851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:02.562057 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:02.562520 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:02.562877 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:02.564897 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:02.566355 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:02.568218 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:02.569093 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:02.569259 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:02.574453 (+ 5194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:02.574468 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:02.574497 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:02.574556 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:02.578066 (+ 3510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:02.579075 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:02.579084 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:02.579091 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:02.579299 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:02.579753 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:02.579764 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:02.579766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:02.580370 (+ 604us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:02.580610 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:02.581116 (+ 506us) server_negotiation.cc:299] Negotiation successful
0926 16:31:02.581546 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":674,"thread_start_us":418,"threads_started":1}
I20260926 16:31:02.586722 9275 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9df010c358234af3923f19c6bed4beb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9df010c358234af3923f19c6bed4beb1" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 35479 } }
I20260926 16:31:02.587157 6276 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:31:02.587177 9275 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9df010c358234af3923f19c6bed4beb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:31:02.587282 9275 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9df010c358234af3923f19c6bed4beb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9df010c358234af3923f19c6bed4beb1, State: Initialized, Role: FOLLOWER
I20260926 16:31:02.587857 9275 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9df010c358234af3923f19c6bed4beb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9df010c358234af3923f19c6bed4beb1" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 35479 } }
I20260926 16:31:02.588078 9275 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9df010c358234af3923f19c6bed4beb1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:31:02.588202 9275 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9df010c358234af3923f19c6bed4beb1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:31:02.588372 9275 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9df010c358234af3923f19c6bed4beb1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:31:02.590642 9275 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9df010c358234af3923f19c6bed4beb1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9df010c358234af3923f19c6bed4beb1" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 35479 } }
I20260926 16:31:02.591152 9275 leader_election.cc:302] T 00000000000000000000000000000000 P 9df010c358234af3923f19c6bed4beb1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9df010c358234af3923f19c6bed4beb1; no voters:
I20260926 16:31:02.591804 9275 leader_election.cc:288] T 00000000000000000000000000000000 P 9df010c358234af3923f19c6bed4beb1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:31:02.591908 9280 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9df010c358234af3923f19c6bed4beb1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:31:02.593465 9280 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9df010c358234af3923f19c6bed4beb1 [term 1 LEADER]: Becoming Leader. State: Replica: 9df010c358234af3923f19c6bed4beb1, State: Running, Role: LEADER
I20260926 16:31:02.594048 9280 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9df010c358234af3923f19c6bed4beb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9df010c358234af3923f19c6bed4beb1" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 35479 } }
I20260926 16:31:02.594326 9275 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9df010c358234af3923f19c6bed4beb1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:31:02.601328 9282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9df010c358234af3923f19c6bed4beb1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9df010c358234af3923f19c6bed4beb1. Latest consensus state: current_term: 1 leader_uuid: "9df010c358234af3923f19c6bed4beb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9df010c358234af3923f19c6bed4beb1" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 35479 } } }
I20260926 16:31:02.601984 9282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9df010c358234af3923f19c6bed4beb1 [sys.catalog]: This master's current role is: LEADER
I20260926 16:31:02.609330 9281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9df010c358234af3923f19c6bed4beb1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9df010c358234af3923f19c6bed4beb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9df010c358234af3923f19c6bed4beb1" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 35479 } } }
I20260926 16:31:02.609822 9281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9df010c358234af3923f19c6bed4beb1 [sys.catalog]: This master's current role is: LEADER
I20260926 16:31:02.613492 9290 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:31:02.619540 9290 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:31:02.634218 9290 catalog_manager.cc:1347] Generated new cluster ID: 0b7c36923e3d4f32a38af9ab9c2fc456
I20260926 16:31:02.634347 9290 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:31:02.642238 9278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:02.632053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60336 (local address 127.6.33.62:35479)
0926 16:31:02.632244 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:02.632250 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:02.632866 (+ 616us) server_negotiation.cc:407] Connection header received
0926 16:31:02.633093 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:02.633100 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:02.633189 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:02.633322 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:02.634656 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:02.635800 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:02.637769 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:02.638407 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:02.638554 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:02.639834 (+ 1280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:02.639841 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:02.639846 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:31:02.640742 (+ 896us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:02.640887 (+ 145us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:31:02.640916 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:02.641035 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:02.641192 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 16:31:02.641287 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 16:31:02.642953 9290 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:31:02.644534 9290 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:31:02.652372 9290 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9df010c358234af3923f19c6bed4beb1: Generated new TSK 0
I20260926 16:31:02.653568 9290 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:31:02.670037 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 9208
I20260926 16:31:02.671479 9270 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:31:02.762069 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 9208
2026-09-26T16:31:02Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1953 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:31:02 dist-test-slave-khd3 krb5kdc[9310](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:31:02 dist-test-slave-khd3 krb5kdc[9310](info): set up 2 sockets
Sep 26 16:31:02 dist-test-slave-khd3 krb5kdc[9310](info): commencing operation
krb5kdc: starting...
W20260926 16:31:03.922286 6276 mini_kdc.cc:121] Time spent starting KDC: real 1.124s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Sep 26 16:31:04 dist-test-slave-khd3 krb5kdc[9310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:31:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:31:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab.
Entry for principal kudu/127.6.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab.
I20260926 16:31:04.120927 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:34851
--webserver_interface=127.6.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:46111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.33.62:34851
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:31:04.438637 9323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:31:04.438916 9323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:31:04.439087 9323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:31:04.446458 9323 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:31:04.446650 9323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:31:04.446738 9323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:31:04.446842 9323 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:31:04.446905 9323 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:31:04.456656 9323 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:46111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.33.62:34851
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:34851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:31:04.458019 9323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:31:04.459335 9323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:31:04.471035 9328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:04.470997 9331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:04.471068 9329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:31:04.472359 9323 server_base.cc:1034] running on GCE node
I20260926 16:31:04.473238 9323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:31:04.474689 9323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:31:04.475924 9323 hybrid_clock.cc:648] HybridClock initialized: now 1790440264475889 us; error 57 us; skew 500 ppm
I20260926 16:31:04.476289 9323 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:31:04 dist-test-slave-khd3 krb5kdc[9310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440264, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:31:04.479561 9323 init.cc:377] Logged in from keytab as kudu/127.6.33.62@KRBTEST.COM (short username kudu)
I20260926 16:31:04.481163 9323 webserver.cc:460] Webserver started at http://127.6.33.62:46713/ using document root <none> and password file <none>
I20260926 16:31:04.481853 9323 fs_manager.cc:359] Metadata directory not provided
I20260926 16:31:04.481995 9323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:31:04.482298 9323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:31:04.485307 9323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/minicluster-data/master-0/data/instance:
uuid: "67cb754fc8ee406eb0149fe47b877e10"
format_stamp: "Formatted at 2026-09-26 16:31:04 on dist-test-slave-khd3"
server_key: "8080417e7c2def72102210bbdaa01e1e"
server_key_iv: "10b34ec02689875c1fe841cba5ebcbc7"
server_key_version: "encryptionkey@0"
I20260926 16:31:04.486058 9323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/minicluster-data/master-0/wal/instance:
uuid: "67cb754fc8ee406eb0149fe47b877e10"
format_stamp: "Formatted at 2026-09-26 16:31:04 on dist-test-slave-khd3"
server_key: "8080417e7c2def72102210bbdaa01e1e"
server_key_iv: "10b34ec02689875c1fe841cba5ebcbc7"
server_key_version: "encryptionkey@0"
I20260926 16:31:04.491080 9323 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 16:31:04.494293 9338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:04.495507 9323 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 16:31:04.495709 9323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/minicluster-data/master-0/wal
uuid: "67cb754fc8ee406eb0149fe47b877e10"
format_stamp: "Formatted at 2026-09-26 16:31:04 on dist-test-slave-khd3"
server_key: "8080417e7c2def72102210bbdaa01e1e"
server_key_iv: "10b34ec02689875c1fe841cba5ebcbc7"
server_key_version: "encryptionkey@0"
I20260926 16:31:04.495893 9323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:31:04.516076 9323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:31:04.520107 9323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:31:04.520395 9323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:31:04.538200 9323 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.62:34851
I20260926 16:31:04.538249 9389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.62:34851 every 8 connection(s)
I20260926 16:31:04.539626 9323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
I20260926 16:31:04.543311 9390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:31:04.548911 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 9323
I20260926 16:31:04.549153 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/minicluster-data/master-0/wal/instance
I20260926 16:31:04.549746 6276 external_mini_cluster.cc:1428] Setting key aaaa6b545607c5583a083a91f08a3434
I20260926 16:31:04.554018 9390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67cb754fc8ee406eb0149fe47b877e10: Bootstrap starting.
I20260926 16:31:04.557706 9390 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67cb754fc8ee406eb0149fe47b877e10: Neither blocks nor log segments found. Creating new log.
I20260926 16:31:04.558915 9390 log.cc:824] T 00000000000000000000000000000000 P 67cb754fc8ee406eb0149fe47b877e10: Log is configured to *not* fsync() on all Append() calls
I20260926 16:31:04.561892 9390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67cb754fc8ee406eb0149fe47b877e10: No bootstrap required, opened a new log
Sep 26 16:31:04 dist-test-slave-khd3 krb5kdc[9310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:31:04.570078 9390 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67cb754fc8ee406eb0149fe47b877e10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67cb754fc8ee406eb0149fe47b877e10" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 34851 } }
I20260926 16:31:04.570472 9390 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67cb754fc8ee406eb0149fe47b877e10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:31:04.570606 9390 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67cb754fc8ee406eb0149fe47b877e10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67cb754fc8ee406eb0149fe47b877e10, State: Initialized, Role: FOLLOWER
I20260926 16:31:04.571317 9390 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67cb754fc8ee406eb0149fe47b877e10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67cb754fc8ee406eb0149fe47b877e10" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 34851 } }
I20260926 16:31:04.571593 9390 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67cb754fc8ee406eb0149fe47b877e10 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:31:04.571745 9390 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67cb754fc8ee406eb0149fe47b877e10 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:31:04.571941 9390 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67cb754fc8ee406eb0149fe47b877e10 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:31:04.572678 9393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:04.551726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47610 (local address 127.6.33.62:34851)
0926 16:31:04.552538 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:04.552550 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:04.552593 (+ 43us) server_negotiation.cc:407] Connection header received
0926 16:31:04.554337 (+ 1744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:04.554358 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:04.554751 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:04.555079 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:04.556860 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:04.558202 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:04.559915 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:04.560577 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:04.560813 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:04.564889 (+ 4076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:04.564897 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:04.564915 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:04.564959 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:04.568013 (+ 3054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:04.568747 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:04.568794 (+ 47us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:04.568805 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:04.568984 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:04.569380 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:04.569386 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:04.569389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:04.569941 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:04.570258 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:04.570746 (+ 488us) server_negotiation.cc:299] Negotiation successful
0926 16:31:04.571189 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":401,"threads_started":1}
I20260926 16:31:04.574972 9390 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67cb754fc8ee406eb0149fe47b877e10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67cb754fc8ee406eb0149fe47b877e10" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 34851 } }
I20260926 16:31:04.575549 9390 leader_election.cc:302] T 00000000000000000000000000000000 P 67cb754fc8ee406eb0149fe47b877e10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67cb754fc8ee406eb0149fe47b877e10; no voters:
I20260926 16:31:04.576122 9390 leader_election.cc:288] T 00000000000000000000000000000000 P 67cb754fc8ee406eb0149fe47b877e10 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:31:04.576253 9395 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67cb754fc8ee406eb0149fe47b877e10 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:31:04.576962 9395 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67cb754fc8ee406eb0149fe47b877e10 [term 1 LEADER]: Becoming Leader. State: Replica: 67cb754fc8ee406eb0149fe47b877e10, State: Running, Role: LEADER
I20260926 16:31:04.577600 9395 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67cb754fc8ee406eb0149fe47b877e10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67cb754fc8ee406eb0149fe47b877e10" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 34851 } }
I20260926 16:31:04.578341 9390 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67cb754fc8ee406eb0149fe47b877e10 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:31:04.580474 9397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67cb754fc8ee406eb0149fe47b877e10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67cb754fc8ee406eb0149fe47b877e10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67cb754fc8ee406eb0149fe47b877e10" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 34851 } } }
I20260926 16:31:04.580844 9397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67cb754fc8ee406eb0149fe47b877e10 [sys.catalog]: This master's current role is: LEADER
I20260926 16:31:04.581300 9396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67cb754fc8ee406eb0149fe47b877e10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67cb754fc8ee406eb0149fe47b877e10. Latest consensus state: current_term: 1 leader_uuid: "67cb754fc8ee406eb0149fe47b877e10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67cb754fc8ee406eb0149fe47b877e10" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 34851 } } }
I20260926 16:31:04.581632 9396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67cb754fc8ee406eb0149fe47b877e10 [sys.catalog]: This master's current role is: LEADER
I20260926 16:31:04.582711 9402 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:31:04.587322 9402 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:31:04.595767 9402 catalog_manager.cc:1347] Generated new cluster ID: dffa9277aca445e59e78b7b19bf615a9
I20260926 16:31:04.595896 9402 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:31:04.613454 9402 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:31:04.614797 9402 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:31:04.627043 9402 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67cb754fc8ee406eb0149fe47b877e10: Generated new TSK 0
I20260926 16:31:04.627913 9402 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.33.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab.
Entry for principal kudu/127.6.33.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab.
I20260926 16:31:04.672574 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.1:0
--local_ip_for_outbound_sockets=127.6.33.1
--webserver_interface=127.6.33.1
--webserver_port=0
--tserver_master_addrs=127.6.33.62:34851
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab
--principal=kudu/127.6.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:46111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:31:05.022378 9416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:31:05.022681 9416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:31:05.022843 9416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:31:05.030171 9416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:31:05.030282 9416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:31:05.030471 9416 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.1
I20260926 16:31:05.040499 9416 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:46111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab
--principal=kudu/127.6.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.33.1
--webserver_port=0
--tserver_master_addrs=127.6.33.62:34851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.1
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:31:05.041944 9416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:31:05.043244 9416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:31:05.056362 9424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:05.056442 9421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:05.056424 9422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:31:05.057775 9416 server_base.cc:1034] running on GCE node
I20260926 16:31:05.058704 9416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:31:05.059733 9416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:31:05.061012 9416 hybrid_clock.cc:648] HybridClock initialized: now 1790440265060968 us; error 54 us; skew 500 ppm
I20260926 16:31:05.061378 9416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:31:05 dist-test-slave-khd3 krb5kdc[9310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440265, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:31:05.064893 9416 init.cc:377] Logged in from keytab as kudu/127.6.33.1@KRBTEST.COM (short username kudu)
I20260926 16:31:05.066437 9416 webserver.cc:460] Webserver started at http://127.6.33.1:39979/ using document root <none> and password file <none>
I20260926 16:31:05.067162 9416 fs_manager.cc:359] Metadata directory not provided
I20260926 16:31:05.067307 9416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:31:05.067613 9416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:31:05.070636 9416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-0/data/instance:
uuid: "b4c66bb4449d4b098ea67dbb6266b38b"
format_stamp: "Formatted at 2026-09-26 16:31:05 on dist-test-slave-khd3"
server_key: "9df5e4834f66682df9102b1a3e8aa85a"
server_key_iv: "65415c685f3ca9c7b29540132a439c86"
server_key_version: "encryptionkey@0"
I20260926 16:31:05.071538 9416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-0/wal/instance:
uuid: "b4c66bb4449d4b098ea67dbb6266b38b"
format_stamp: "Formatted at 2026-09-26 16:31:05 on dist-test-slave-khd3"
server_key: "9df5e4834f66682df9102b1a3e8aa85a"
server_key_iv: "65415c685f3ca9c7b29540132a439c86"
server_key_version: "encryptionkey@0"
I20260926 16:31:05.077104 9416 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260926 16:31:05.080607 9431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:05.082062 9416 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:31:05.082247 9416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-0/wal
uuid: "b4c66bb4449d4b098ea67dbb6266b38b"
format_stamp: "Formatted at 2026-09-26 16:31:05 on dist-test-slave-khd3"
server_key: "9df5e4834f66682df9102b1a3e8aa85a"
server_key_iv: "65415c685f3ca9c7b29540132a439c86"
server_key_version: "encryptionkey@0"
I20260926 16:31:05.082451 9416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:31:05.103662 9416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:31:05.108124 9416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:31:05.108428 9416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:31:05.109455 9416 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:31:05.110818 9416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:31:05.110926 9416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:05.111068 9416 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:31:05.111153 9416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:05.144529 9416 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.1:43537
I20260926 16:31:05.144629 9543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.1:43537 every 8 connection(s)
I20260926 16:31:05.145941 9416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
I20260926 16:31:05.153667 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 9416
I20260926 16:31:05.153908 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-0/wal/instance
I20260926 16:31:05.154695 6276 external_mini_cluster.cc:1428] Setting key b7dfcea9654c4207d33a013014a08270
Sep 26 16:31:05 dist-test-slave-khd3 krb5kdc[9310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440265, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.1@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:31:05.167599 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:05.148646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:39659 (local address 127.6.33.62:34851)
0926 16:31:05.149118 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:05.149124 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:05.150071 (+ 947us) server_negotiation.cc:407] Connection header received
0926 16:31:05.151867 (+ 1796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:05.151871 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:05.151937 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:05.152033 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:05.153870 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:05.155004 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:05.156946 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:05.157585 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:05.157729 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:05.161629 (+ 3900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:05.161634 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:05.161638 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:05.161670 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:05.164271 (+ 2601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:05.165274 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:05.165278 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:05.165280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:05.165395 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:05.165889 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:05.165892 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:05.165894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:05.166151 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:05.166258 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:05.167188 (+ 930us) server_negotiation.cc:299] Negotiation successful
0926 16:31:05.167349 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":235,"threads_started":1}
I20260926 16:31:05.169291 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:05.149215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:34851 (local address 127.6.33.1:39659)
0926 16:31:05.149914 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:05.149951 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:05.151592 (+ 1641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:05.152244 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:05.152252 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:05.153043 (+ 791us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:05.153644 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:05.153666 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:05.155186 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:05.155190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:05.156740 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:05.156801 (+ 61us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:05.157817 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:05.157823 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:05.158035 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:05.158874 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:05.158898 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:05.161343 (+ 2445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:31:05.164420 (+ 3077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:05.164433 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:05.164453 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:05.165147 (+ 694us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:05.165502 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:05.165510 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:05.165516 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:05.165767 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:05.166284 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:31:05.166296 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:31:05.166767 (+ 471us) client_negotiation.cc:769] Sending connection context
0926 16:31:05.167069 (+ 302us) client_negotiation.cc:240] Negotiation successful
0926 16:31:05.167686 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":266,"threads_started":1}
I20260926 16:31:05.171176 9544 heartbeater.cc:347] Connected to a master server at 127.6.33.62:34851
I20260926 16:31:05.171567 9544 heartbeater.cc:464] Registering TS with master...
I20260926 16:31:05.172366 9544 heartbeater.cc:511] Master 127.6.33.62:34851 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.33.2@KRBTEST.COM; defaulting to no policy
I20260926 16:31:05.174769 9355 ts_manager.cc:194] Registered new tserver with Master: b4c66bb4449d4b098ea67dbb6266b38b (127.6.33.1:43537)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.2@KRBTEST.COM" created.
I20260926 16:31:05.176306 9355 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.33.1@KRBTEST.COM'} at 127.6.33.1:39659
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab.
Entry for principal kudu/127.6.33.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab.
I20260926 16:31:05.200277 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.2:0
--local_ip_for_outbound_sockets=127.6.33.2
--webserver_interface=127.6.33.2
--webserver_port=0
--tserver_master_addrs=127.6.33.62:34851
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab
--principal=kudu/127.6.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:46111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:31:05.521512 9550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:31:05.521867 9550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:31:05.522050 9550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:31:05.529886 9550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:31:05.530037 9550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:31:05.530205 9550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.2
I20260926 16:31:05.541883 9550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:46111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab
--principal=kudu/127.6.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.33.2
--webserver_port=0
--tserver_master_addrs=127.6.33.62:34851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.2
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:31:05.543458 9550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:31:05.545193 9550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:31:05.560261 9555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:05.560283 9556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:05.561137 9558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:31:05.561321 9550 server_base.cc:1034] running on GCE node
I20260926 16:31:05.562005 9550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:31:05.562994 9550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:31:05.564244 9550 hybrid_clock.cc:648] HybridClock initialized: now 1790440265564158 us; error 55 us; skew 500 ppm
I20260926 16:31:05.564581 9550 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:31:05 dist-test-slave-khd3 krb5kdc[9310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440265, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:31:05.567818 9550 init.cc:377] Logged in from keytab as kudu/127.6.33.2@KRBTEST.COM (short username kudu)
I20260926 16:31:05.569547 9550 webserver.cc:460] Webserver started at http://127.6.33.2:37355/ using document root <none> and password file <none>
I20260926 16:31:05.570283 9550 fs_manager.cc:359] Metadata directory not provided
I20260926 16:31:05.570430 9550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:31:05.570756 9550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:31:05.573833 9550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-1/data/instance:
uuid: "f7a546204ac9433cae036c481e25b20f"
format_stamp: "Formatted at 2026-09-26 16:31:05 on dist-test-slave-khd3"
server_key: "ac3372e2967a8e4be231b3054c6ba254"
server_key_iv: "c473341bc70df897f01a6a3bf9e0e2b7"
server_key_version: "encryptionkey@0"
I20260926 16:31:05.574644 9550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-1/wal/instance:
uuid: "f7a546204ac9433cae036c481e25b20f"
format_stamp: "Formatted at 2026-09-26 16:31:05 on dist-test-slave-khd3"
server_key: "ac3372e2967a8e4be231b3054c6ba254"
server_key_iv: "c473341bc70df897f01a6a3bf9e0e2b7"
server_key_version: "encryptionkey@0"
I20260926 16:31:05.580303 9550 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 16:31:05.583928 9565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:05.585448 9550 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 16:31:05.585614 9550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-1/wal
uuid: "f7a546204ac9433cae036c481e25b20f"
format_stamp: "Formatted at 2026-09-26 16:31:05 on dist-test-slave-khd3"
server_key: "ac3372e2967a8e4be231b3054c6ba254"
server_key_iv: "c473341bc70df897f01a6a3bf9e0e2b7"
server_key_version: "encryptionkey@0"
I20260926 16:31:05.585862 9550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:31:05.606715 9550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:31:05.610946 9550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:31:05.611275 9550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:31:05.612314 9550 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:31:05.613679 9550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:31:05.613754 9550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:05.613901 9550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:31:05.613953 9550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:05.651525 9550 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.2:42405
I20260926 16:31:05.651530 9677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.2:42405 every 8 connection(s)
I20260926 16:31:05.652904 9550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
I20260926 16:31:05.658860 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 9550
I20260926 16:31:05.659076 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-1/wal/instance
I20260926 16:31:05.659607 6276 external_mini_cluster.cc:1428] Setting key 861958c8bc50a461c81b992f6641887e
Sep 26 16:31:05 dist-test-slave-khd3 krb5kdc[9310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440265, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.2@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:31:05.675181 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:05.655535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:48205 (local address 127.6.33.62:34851)
0926 16:31:05.655719 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:05.655724 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:05.656991 (+ 1267us) server_negotiation.cc:407] Connection header received
0926 16:31:05.659040 (+ 2049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:05.659045 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:05.659166 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:05.659268 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:05.661204 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:05.662146 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:05.664853 (+ 2707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:05.665286 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:05.665393 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:05.669522 (+ 4129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:05.669527 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:05.669530 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:05.669563 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:05.672068 (+ 2505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:05.672813 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:05.672823 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:05.672825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:05.672944 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:05.673383 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:05.673388 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:05.673391 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:05.673646 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:05.673796 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:05.674598 (+ 802us) server_negotiation.cc:299] Negotiation successful
0926 16:31:05.674829 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 16:31:05.676168 9680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:05.656074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:34851 (local address 127.6.33.2:48205)
0926 16:31:05.656849 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:05.656887 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:05.658651 (+ 1764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:05.659551 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:05.659563 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:05.660280 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:05.661042 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:05.661062 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:05.662309 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:05.662315 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:05.664621 (+ 2306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:05.664640 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:05.665528 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:05.665538 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:05.665887 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:05.666692 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:05.666725 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:05.669289 (+ 2564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:31:05.672228 (+ 2939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:05.672238 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:05.672255 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:05.672662 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:05.673052 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:05.673058 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:05.673062 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:05.673267 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:05.673770 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:31:05.673778 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:31:05.674160 (+ 382us) client_negotiation.cc:769] Sending connection context
0926 16:31:05.674496 (+ 336us) client_negotiation.cc:240] Negotiation successful
0926 16:31:05.675046 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":266,"threads_started":1}
I20260926 16:31:05.677850 9678 heartbeater.cc:347] Connected to a master server at 127.6.33.62:34851
I20260926 16:31:05.678157 9678 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.6.33.3@KRBTEST.COM; defaulting to no policy
I20260926 16:31:05.678975 9678 heartbeater.cc:511] Master 127.6.33.62:34851 requested a full tablet report, sending...
I20260926 16:31:05.680622 9355 ts_manager.cc:194] Registered new tserver with Master: f7a546204ac9433cae036c481e25b20f (127.6.33.2:42405)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.3@KRBTEST.COM" created.
I20260926 16:31:05.681658 9355 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.33.2@KRBTEST.COM'} at 127.6.33.2:48205
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab.
Entry for principal kudu/127.6.33.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab.
I20260926 16:31:05.706561 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.3:0
--local_ip_for_outbound_sockets=127.6.33.3
--webserver_interface=127.6.33.3
--webserver_port=0
--tserver_master_addrs=127.6.33.62:34851
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab
--principal=kudu/127.6.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:46111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:31:06.028491 9683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:31:06.028889 9683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:31:06.029059 9683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:31:06.036474 9683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:31:06.036590 9683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:31:06.036823 9683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.3
I20260926 16:31:06.046746 9683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:46111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab
--principal=kudu/127.6.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.33.3
--webserver_port=0
--tserver_master_addrs=127.6.33.62:34851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.3
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:31:06.048101 9683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:31:06.049559 9683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:31:06.063686 9691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:06.063747 9688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:06.063722 9689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:31:06.064807 9683 server_base.cc:1034] running on GCE node
I20260926 16:31:06.065491 9683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:31:06.066459 9683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:31:06.067719 9683 hybrid_clock.cc:648] HybridClock initialized: now 1790440266067659 us; error 44 us; skew 500 ppm
I20260926 16:31:06.068075 9683 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:31:06 dist-test-slave-khd3 krb5kdc[9310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440266, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:31:06.071529 9683 init.cc:377] Logged in from keytab as kudu/127.6.33.3@KRBTEST.COM (short username kudu)
I20260926 16:31:06.073022 9683 webserver.cc:460] Webserver started at http://127.6.33.3:43725/ using document root <none> and password file <none>
I20260926 16:31:06.073738 9683 fs_manager.cc:359] Metadata directory not provided
I20260926 16:31:06.073869 9683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:31:06.074190 9683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:31:06.077252 9683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-2/data/instance:
uuid: "c671986e64474bcba1180dd4acfad27f"
format_stamp: "Formatted at 2026-09-26 16:31:06 on dist-test-slave-khd3"
server_key: "004c513a77978b8a2c55c9d16aca4ecb"
server_key_iv: "0a3487159e4e542caf76c10a541eaa9e"
server_key_version: "encryptionkey@0"
I20260926 16:31:06.078086 9683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-2/wal/instance:
uuid: "c671986e64474bcba1180dd4acfad27f"
format_stamp: "Formatted at 2026-09-26 16:31:06 on dist-test-slave-khd3"
server_key: "004c513a77978b8a2c55c9d16aca4ecb"
server_key_iv: "0a3487159e4e542caf76c10a541eaa9e"
server_key_version: "encryptionkey@0"
I20260926 16:31:06.083680 9683 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 16:31:06.087482 9698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:06.088871 9683 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:31:06.089039 9683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-2/wal
uuid: "c671986e64474bcba1180dd4acfad27f"
format_stamp: "Formatted at 2026-09-26 16:31:06 on dist-test-slave-khd3"
server_key: "004c513a77978b8a2c55c9d16aca4ecb"
server_key_iv: "0a3487159e4e542caf76c10a541eaa9e"
server_key_version: "encryptionkey@0"
I20260926 16:31:06.089202 9683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:31:06.110091 9683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:31:06.114427 9683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:31:06.114743 9683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:31:06.115724 9683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:31:06.117116 9683 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:31:06.117194 9683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:06.117336 9683 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:31:06.117394 9683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:06.150760 9683 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.3:35029
I20260926 16:31:06.150874 9810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.3:35029 every 8 connection(s)
I20260926 16:31:06.152076 9683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
I20260926 16:31:06.154808 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 9683
I20260926 16:31:06.155011 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440233850748-6276-0/minicluster-data/ts-2/wal/instance
I20260926 16:31:06.155485 6276 external_mini_cluster.cc:1428] Setting key 2a667b105dbda1a0067fe3fb40e064e1
Sep 26 16:31:06 dist-test-slave-khd3 krb5kdc[9310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440266, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.3@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:31:06.171649 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:06.154723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.3:35107 (local address 127.6.33.62:34851)
0926 16:31:06.154957 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:06.154964 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:06.156170 (+ 1206us) server_negotiation.cc:407] Connection header received
0926 16:31:06.158066 (+ 1896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:06.158073 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:06.158171 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:06.158259 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:06.159746 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:06.160519 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:06.162093 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:06.162495 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:06.162603 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:06.165825 (+ 3222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:06.165829 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:06.165832 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:06.165871 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:06.168493 (+ 2622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:06.169326 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:06.169333 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:06.169337 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:06.169502 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:06.170012 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:06.170015 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:06.170017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:06.170370 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:06.170510 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:06.171243 (+ 733us) server_negotiation.cc:299] Negotiation successful
0926 16:31:06.171397 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 16:31:06.172590 9813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:06.155208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:34851 (local address 127.6.33.3:35107)
0926 16:31:06.156007 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:06.156060 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:06.157817 (+ 1757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:06.158528 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:06.158537 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:06.159027 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:06.159587 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:06.159603 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:06.160633 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:06.160637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:06.161945 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:06.161956 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:06.162671 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:06.162675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:06.162897 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:06.163507 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:06.163528 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:06.165625 (+ 2097us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:31:06.168599 (+ 2974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:06.168605 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:06.168617 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:06.169210 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:06.169626 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:06.169630 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:06.169634 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:06.169858 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:06.170501 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:31:06.170511 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:31:06.170909 (+ 398us) client_negotiation.cc:769] Sending connection context
0926 16:31:06.171128 (+ 219us) client_negotiation.cc:240] Negotiation successful
0926 16:31:06.171502 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":327,"threads_started":1}
I20260926 16:31:06.173918 9811 heartbeater.cc:347] Connected to a master server at 127.6.33.62:34851
I20260926 16:31:06.174213 9811 heartbeater.cc:464] Registering TS with master...
I20260926 16:31:06.174899 9811 heartbeater.cc:511] Master 127.6.33.62:34851 requested a full tablet report, sending...
I20260926 16:31:06.176270 9355 ts_manager.cc:194] Registered new tserver with Master: c671986e64474bcba1180dd4acfad27f (127.6.33.3:35029)
I20260926 16:31:06.177136 9355 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.33.3@KRBTEST.COM'} at 127.6.33.3:35107
I20260926 16:31:06.179504 9544 heartbeater.cc:503] Master 127.6.33.62:34851 was elected leader, sending a full tablet report...
I20260926 16:31:06.183284 6276 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 16:31:06 dist-test-slave-khd3 krb5kdc[9310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440266, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 26 16:31:06 dist-test-slave-khd3 krb5kdc[9310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440266, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:31:06.231340 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:06.217668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47612 (local address 127.6.33.62:34851)
0926 16:31:06.217898 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:06.217906 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:06.218297 (+ 391us) server_negotiation.cc:407] Connection header received
0926 16:31:06.218524 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:06.218528 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:06.218590 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:06.218701 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:06.219995 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:06.220696 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:06.221921 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:06.222325 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:06.222460 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:06.226338 (+ 3878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:06.226344 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:06.226348 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:06.226382 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:06.229038 (+ 2656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:06.229679 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:06.229683 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:06.229685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:06.229799 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:06.230189 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:06.230193 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:06.230195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:06.230519 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:06.230715 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:06.230939 (+ 224us) server_negotiation.cc:299] Negotiation successful
0926 16:31:06.231112 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260926 16:31:06.231951 9355 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:47612
I20260926 16:31:06.234082 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 9416
Sep 26 16:31:06 dist-test-slave-khd3 krb5kdc[9310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440266, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.6.33.1@KRBTEST.COM
I20260926 16:31:06.249543 9822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:06.235059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44464 (local address 127.6.33.1:43537)
0926 16:31:06.235671 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:06.235678 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:06.235705 (+ 27us) server_negotiation.cc:407] Connection header received
0926 16:31:06.235759 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:06.235766 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:06.235968 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:06.236103 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:06.237327 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:06.238352 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:06.239568 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:06.240111 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:06.240212 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:06.244213 (+ 4001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:06.244225 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:06.244240 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:06.244274 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:06.247214 (+ 2940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:06.247821 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:06.247827 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:06.247829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:06.247944 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:06.248255 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:06.248258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:06.248260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:06.248597 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:06.248869 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:06.249075 (+ 206us) server_negotiation.cc:299] Negotiation successful
0926 16:31:06.249265 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":329,"threads_started":1}
W20260926 16:31:06.250909 9539 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:44464
W20260926 16:31:06.252028 6276 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 9416: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 16:31:06.252234 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 9416
I20260926 16:31:06.267704 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 9550
Sep 26 16:31:06 dist-test-slave-khd3 krb5kdc[9310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440266, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.6.33.2@KRBTEST.COM
I20260926 16:31:06.283627 9823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:06.268608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52280 (local address 127.6.33.2:42405)
0926 16:31:06.269117 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:06.269124 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:06.269150 (+ 26us) server_negotiation.cc:407] Connection header received
0926 16:31:06.269213 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:06.269220 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:06.269427 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:06.269548 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:06.270743 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:06.271704 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:06.272976 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:06.273553 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:06.273648 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:06.278044 (+ 4396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:06.278057 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:06.278072 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:06.278104 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:06.281131 (+ 3027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:06.281730 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:06.281737 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:06.281739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:06.281856 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:06.282150 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:06.282154 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:06.282155 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:06.282463 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:06.282715 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:06.283011 (+ 296us) server_negotiation.cc:299] Negotiation successful
0926 16:31:06.283211 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":268,"threads_started":1}
W20260926 16:31:06.284966 9673 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52280
W20260926 16:31:06.286017 6276 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 9550: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 16:31:06.286170 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 9550
I20260926 16:31:06.302515 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 9683
Sep 26 16:31:06 dist-test-slave-khd3 krb5kdc[9310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440266, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.6.33.3@KRBTEST.COM
I20260926 16:31:06.319298 9824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:06.303416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33552 (local address 127.6.33.3:35029)
0926 16:31:06.304049 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:06.304057 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:06.304085 (+ 28us) server_negotiation.cc:407] Connection header received
0926 16:31:06.304150 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:06.304157 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:06.304366 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:06.304501 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:06.305826 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:06.306884 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:06.308134 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:06.308875 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:06.308992 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:06.313203 (+ 4211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:06.313216 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:06.313230 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:06.313265 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:06.316173 (+ 2908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:06.316859 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:06.316867 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:06.316869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:06.317002 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:06.317368 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:06.317372 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:06.317374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:06.317759 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:06.317987 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:06.318269 (+ 282us) server_negotiation.cc:299] Negotiation successful
0926 16:31:06.319001 (+ 732us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":350,"threads_started":1}
W20260926 16:31:06.320574 9806 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:33552
W20260926 16:31:06.321774 6276 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 9683: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 16:31:06.321904 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 9683
I20260926 16:31:06.336864 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 9323
I20260926 16:31:06.337985 9385 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:31:06.438367 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 9323
2026-09-26T16:31:06Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3681 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:31:06 dist-test-slave-khd3 krb5kdc[9828](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:31:06 dist-test-slave-khd3 krb5kdc[9828](info): set up 2 sockets
Sep 26 16:31:06 dist-test-slave-khd3 krb5kdc[9828](info): commencing operation
krb5kdc: starting...
W20260926 16:31:07.522250 6276 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Sep 26 16:31:07 dist-test-slave-khd3 krb5kdc[9828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:31:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:31:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab.
Entry for principal kudu/127.6.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab.
I20260926 16:31:07.725095 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:41009
--webserver_interface=127.6.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:38175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.33.62:41009
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:31:08.044698 9841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:31:08.045323 9841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:31:08.045517 9841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:31:08.052780 9841 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:31:08.052888 9841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:31:08.052978 9841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:31:08.053073 9841 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:31:08.053170 9841 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:31:08.063148 9841 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:38175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.33.62:41009
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:41009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:31:08.064509 9841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:31:08.066373 9841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:31:08.078743 9846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:08.078804 9847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:08.078783 9849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:31:08.080229 9841 server_base.cc:1034] running on GCE node
I20260926 16:31:08.081136 9841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:31:08.082595 9841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:31:08.083806 9841 hybrid_clock.cc:648] HybridClock initialized: now 1790440268083764 us; error 35 us; skew 500 ppm
I20260926 16:31:08.084174 9841 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:31:08 dist-test-slave-khd3 krb5kdc[9828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440268, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:31:08.087294 9841 init.cc:377] Logged in from keytab as kudu/127.6.33.62@KRBTEST.COM (short username kudu)
I20260926 16:31:08.088868 9841 webserver.cc:460] Webserver started at http://127.6.33.62:46255/ using document root <none> and password file <none>
I20260926 16:31:08.089540 9841 fs_manager.cc:359] Metadata directory not provided
I20260926 16:31:08.089682 9841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:31:08.089983 9841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:31:08.092885 9841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/minicluster-data/master-0/data/instance:
uuid: "29b6941179c34a0d87dfc1b221129506"
format_stamp: "Formatted at 2026-09-26 16:31:08 on dist-test-slave-khd3"
server_key: "28ac5f5a9e60e1d1e814990a0998b7d0"
server_key_iv: "633e4aededc2118a5f6cdd9d0b94bf11"
server_key_version: "encryptionkey@0"
I20260926 16:31:08.093672 9841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/minicluster-data/master-0/wal/instance:
uuid: "29b6941179c34a0d87dfc1b221129506"
format_stamp: "Formatted at 2026-09-26 16:31:08 on dist-test-slave-khd3"
server_key: "28ac5f5a9e60e1d1e814990a0998b7d0"
server_key_iv: "633e4aededc2118a5f6cdd9d0b94bf11"
server_key_version: "encryptionkey@0"
I20260926 16:31:08.098783 9841 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260926 16:31:08.102313 9856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:08.103665 9841 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 16:31:08.103858 9841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/minicluster-data/master-0/wal
uuid: "29b6941179c34a0d87dfc1b221129506"
format_stamp: "Formatted at 2026-09-26 16:31:08 on dist-test-slave-khd3"
server_key: "28ac5f5a9e60e1d1e814990a0998b7d0"
server_key_iv: "633e4aededc2118a5f6cdd9d0b94bf11"
server_key_version: "encryptionkey@0"
I20260926 16:31:08.104063 9841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:31:08.138958 9841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:31:08.143236 9841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:31:08.143537 9841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:31:08.162796 9841 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.62:41009
I20260926 16:31:08.162829 9907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.62:41009 every 8 connection(s)
I20260926 16:31:08.164239 9841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
I20260926 16:31:08.168262 9908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:31:08.173880 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 9841
I20260926 16:31:08.174124 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/minicluster-data/master-0/wal/instance
I20260926 16:31:08.174772 6276 external_mini_cluster.cc:1428] Setting key 02867570b44acbfbc23eb32023b29dfa
I20260926 16:31:08.180513 9908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29b6941179c34a0d87dfc1b221129506: Bootstrap starting.
I20260926 16:31:08.184579 9908 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29b6941179c34a0d87dfc1b221129506: Neither blocks nor log segments found. Creating new log.
I20260926 16:31:08.185822 9908 log.cc:824] T 00000000000000000000000000000000 P 29b6941179c34a0d87dfc1b221129506: Log is configured to *not* fsync() on all Append() calls
I20260926 16:31:08.189364 9908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29b6941179c34a0d87dfc1b221129506: No bootstrap required, opened a new log
Sep 26 16:31:08 dist-test-slave-khd3 krb5kdc[9828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:31:08.200168 9908 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29b6941179c34a0d87dfc1b221129506 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29b6941179c34a0d87dfc1b221129506" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 41009 } }
I20260926 16:31:08.200644 9908 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29b6941179c34a0d87dfc1b221129506 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:31:08.200771 9908 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29b6941179c34a0d87dfc1b221129506 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b6941179c34a0d87dfc1b221129506, State: Initialized, Role: FOLLOWER
I20260926 16:31:08.201500 9911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:08.177052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58294 (local address 127.6.33.62:41009)
0926 16:31:08.177921 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:08.177939 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:08.178000 (+ 61us) server_negotiation.cc:407] Connection header received
0926 16:31:08.180513 (+ 2513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:08.180546 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:08.181146 (+ 600us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:08.181620 (+ 474us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:08.183087 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:08.185125 (+ 2038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:08.186683 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:08.187671 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:08.187855 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:08.192055 (+ 4200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:08.192065 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:08.192087 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:08.192148 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:08.196359 (+ 4211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:08.197260 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:08.197269 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:08.197278 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:08.197519 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:08.197970 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:08.197975 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:08.197979 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:08.198811 (+ 832us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:08.199135 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:08.199536 (+ 401us) server_negotiation.cc:299] Negotiation successful
0926 16:31:08.200010 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":358,"threads_started":1}
I20260926 16:31:08.201480 9908 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29b6941179c34a0d87dfc1b221129506 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29b6941179c34a0d87dfc1b221129506" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 41009 } }
I20260926 16:31:08.201840 9908 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29b6941179c34a0d87dfc1b221129506 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:31:08.202032 9908 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29b6941179c34a0d87dfc1b221129506 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:31:08.202237 9908 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29b6941179c34a0d87dfc1b221129506 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:31:08.204623 9908 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29b6941179c34a0d87dfc1b221129506 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29b6941179c34a0d87dfc1b221129506" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 41009 } }
I20260926 16:31:08.205205 9908 leader_election.cc:302] T 00000000000000000000000000000000 P 29b6941179c34a0d87dfc1b221129506 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29b6941179c34a0d87dfc1b221129506; no voters:
I20260926 16:31:08.205796 9908 leader_election.cc:288] T 00000000000000000000000000000000 P 29b6941179c34a0d87dfc1b221129506 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:31:08.205957 9913 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29b6941179c34a0d87dfc1b221129506 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:31:08.206686 9913 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29b6941179c34a0d87dfc1b221129506 [term 1 LEADER]: Becoming Leader. State: Replica: 29b6941179c34a0d87dfc1b221129506, State: Running, Role: LEADER
I20260926 16:31:08.207209 9913 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29b6941179c34a0d87dfc1b221129506 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29b6941179c34a0d87dfc1b221129506" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 41009 } }
I20260926 16:31:08.208069 9908 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29b6941179c34a0d87dfc1b221129506 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:31:08.209947 9915 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29b6941179c34a0d87dfc1b221129506 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29b6941179c34a0d87dfc1b221129506. Latest consensus state: current_term: 1 leader_uuid: "29b6941179c34a0d87dfc1b221129506" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29b6941179c34a0d87dfc1b221129506" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 41009 } } }
I20260926 16:31:08.210299 9915 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29b6941179c34a0d87dfc1b221129506 [sys.catalog]: This master's current role is: LEADER
I20260926 16:31:08.210888 9914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29b6941179c34a0d87dfc1b221129506 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29b6941179c34a0d87dfc1b221129506" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29b6941179c34a0d87dfc1b221129506" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 41009 } } }
I20260926 16:31:08.211158 9914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29b6941179c34a0d87dfc1b221129506 [sys.catalog]: This master's current role is: LEADER
I20260926 16:31:08.211293 9919 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:31:08.216621 9919 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:31:08.226651 9919 catalog_manager.cc:1347] Generated new cluster ID: 6d952f723cfd4ac68496ded7370bbbb4
I20260926 16:31:08.226771 9919 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:31:08.239683 9919 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:31:08.241137 9919 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:31:08.248718 9919 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29b6941179c34a0d87dfc1b221129506: Generated new TSK 0
I20260926 16:31:08.249620 9919 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.33.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab.
Entry for principal kudu/127.6.33.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab.
I20260926 16:31:08.304430 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.1:0
--local_ip_for_outbound_sockets=127.6.33.1
--webserver_interface=127.6.33.1
--webserver_port=0
--tserver_master_addrs=127.6.33.62:41009
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab
--principal=kudu/127.6.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:38175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:31:08.630275 9934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:31:08.630558 9934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:31:08.630715 9934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:31:08.638078 9934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:31:08.638201 9934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:31:08.638391 9934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.1
I20260926 16:31:08.648152 9934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:38175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab
--principal=kudu/127.6.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.33.1
--webserver_port=0
--tserver_master_addrs=127.6.33.62:41009
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.1
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:31:08.649600 9934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:31:08.650914 9934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:31:08.664579 9939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:08.664577 9940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:08.664778 9942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:31:08.665731 9934 server_base.cc:1034] running on GCE node
I20260926 16:31:08.666322 9934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:31:08.667232 9934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:31:08.668435 9934 hybrid_clock.cc:648] HybridClock initialized: now 1790440268668358 us; error 79 us; skew 500 ppm
I20260926 16:31:08.668809 9934 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:31:08 dist-test-slave-khd3 krb5kdc[9828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440268, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:31:08.672129 9934 init.cc:377] Logged in from keytab as kudu/127.6.33.1@KRBTEST.COM (short username kudu)
I20260926 16:31:08.673676 9934 webserver.cc:460] Webserver started at http://127.6.33.1:45045/ using document root <none> and password file <none>
I20260926 16:31:08.674391 9934 fs_manager.cc:359] Metadata directory not provided
I20260926 16:31:08.674531 9934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:31:08.674834 9934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:31:08.677774 9934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/minicluster-data/ts-0/data/instance:
uuid: "3ff9d3f7d8784bfb87802aa6bec0bd40"
format_stamp: "Formatted at 2026-09-26 16:31:08 on dist-test-slave-khd3"
server_key: "df81bf079354c1550486386bd2d1a3e5"
server_key_iv: "5e7085f8ec8e3f52b28b274fea153152"
server_key_version: "encryptionkey@0"
I20260926 16:31:08.678589 9934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/minicluster-data/ts-0/wal/instance:
uuid: "3ff9d3f7d8784bfb87802aa6bec0bd40"
format_stamp: "Formatted at 2026-09-26 16:31:08 on dist-test-slave-khd3"
server_key: "df81bf079354c1550486386bd2d1a3e5"
server_key_iv: "5e7085f8ec8e3f52b28b274fea153152"
server_key_version: "encryptionkey@0"
I20260926 16:31:08.683725 9934 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260926 16:31:08.687222 9949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:08.688580 9934 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 16:31:08.688740 9934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/minicluster-data/ts-0/wal
uuid: "3ff9d3f7d8784bfb87802aa6bec0bd40"
format_stamp: "Formatted at 2026-09-26 16:31:08 on dist-test-slave-khd3"
server_key: "df81bf079354c1550486386bd2d1a3e5"
server_key_iv: "5e7085f8ec8e3f52b28b274fea153152"
server_key_version: "encryptionkey@0"
I20260926 16:31:08.689013 9934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:31:08.715961 9934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:31:08.720127 9934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:31:08.720484 9934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:31:08.721439 9934 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:31:08.722719 9934 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:31:08.722815 9934 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:08.722957 9934 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:31:08.723026 9934 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:08.756268 9934 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.1:46107
I20260926 16:31:08.756314 10061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.1:46107 every 8 connection(s)
I20260926 16:31:08.757705 9934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
I20260926 16:31:08.763742 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 9934
I20260926 16:31:08.763944 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/minicluster-data/ts-0/wal/instance
I20260926 16:31:08.764575 6276 external_mini_cluster.cc:1428] Setting key f5ab952db97eeb7f2eac1241f8fb89cf
Sep 26 16:31:08 dist-test-slave-khd3 krb5kdc[9828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440268, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.1@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:31:08.779017 10064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:08.760222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:50953 (local address 127.6.33.62:41009)
0926 16:31:08.760698 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:08.760703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:08.761540 (+ 837us) server_negotiation.cc:407] Connection header received
0926 16:31:08.763345 (+ 1805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:08.763351 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:08.763439 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:08.763555 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:08.765326 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:08.766209 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:08.767876 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:08.768663 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:08.768848 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:08.772994 (+ 4146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:08.772999 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:08.773003 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:08.773035 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:08.775557 (+ 2522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:08.776543 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:08.776547 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:08.776550 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:08.776675 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:08.777180 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:08.777183 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:08.777185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:08.777436 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:08.777543 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:08.778524 (+ 981us) server_negotiation.cc:299] Negotiation successful
0926 16:31:08.778761 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":245,"threads_started":1}
I20260926 16:31:08.780654 10065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:08.760666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:41009 (local address 127.6.33.1:50953)
0926 16:31:08.761391 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:08.761428 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:08.763047 (+ 1619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:08.763789 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:08.763797 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:08.764392 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:08.765125 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:08.765143 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:08.766377 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:08.766382 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:08.767713 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:08.767730 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:08.768963 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:08.768968 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:08.769211 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:08.769929 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:08.769952 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:08.772723 (+ 2771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:31:08.775724 (+ 3001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:08.775735 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:08.775756 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:08.776409 (+ 653us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:08.776807 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:08.776814 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:08.776821 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:08.777058 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:08.777564 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:31:08.777575 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:31:08.778087 (+ 512us) client_negotiation.cc:769] Sending connection context
0926 16:31:08.778423 (+ 336us) client_negotiation.cc:240] Negotiation successful
0926 16:31:08.779063 (+ 640us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":266,"threads_started":1}
I20260926 16:31:08.782490 10062 heartbeater.cc:347] Connected to a master server at 127.6.33.62:41009
I20260926 16:31:08.782806 10062 heartbeater.cc:464] Registering TS with master...
I20260926 16:31:08.783507 10062 heartbeater.cc:511] Master 127.6.33.62:41009 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.33.2@KRBTEST.COM; defaulting to no policy
I20260926 16:31:08.785712 9873 ts_manager.cc:194] Registered new tserver with Master: 3ff9d3f7d8784bfb87802aa6bec0bd40 (127.6.33.1:46107)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.2@KRBTEST.COM" created.
I20260926 16:31:08.787374 9873 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.33.1@KRBTEST.COM'} at 127.6.33.1:50953
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab.
Entry for principal kudu/127.6.33.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab.
I20260926 16:31:08.811561 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.2:0
--local_ip_for_outbound_sockets=127.6.33.2
--webserver_interface=127.6.33.2
--webserver_port=0
--tserver_master_addrs=127.6.33.62:41009
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab
--principal=kudu/127.6.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:38175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:31:09.136157 10068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:31:09.136444 10068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:31:09.136613 10068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:31:09.143877 10068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:31:09.143987 10068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:31:09.144165 10068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.2
I20260926 16:31:09.154126 10068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:38175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab
--principal=kudu/127.6.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.33.2
--webserver_port=0
--tserver_master_addrs=127.6.33.62:41009
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.2
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:31:09.155539 10068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:31:09.157001 10068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:31:09.173020 10074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:09.173171 10073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:09.173151 10076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:31:09.173960 10068 server_base.cc:1034] running on GCE node
I20260926 16:31:09.174606 10068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:31:09.175588 10068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:31:09.176844 10068 hybrid_clock.cc:648] HybridClock initialized: now 1790440269176784 us; error 53 us; skew 500 ppm
I20260926 16:31:09.177160 10068 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:31:09 dist-test-slave-khd3 krb5kdc[9828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440269, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:31:09.180474 10068 init.cc:377] Logged in from keytab as kudu/127.6.33.2@KRBTEST.COM (short username kudu)
I20260926 16:31:09.181974 10068 webserver.cc:460] Webserver started at http://127.6.33.2:39551/ using document root <none> and password file <none>
I20260926 16:31:09.182715 10068 fs_manager.cc:359] Metadata directory not provided
I20260926 16:31:09.182864 10068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:31:09.183190 10068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:31:09.186293 10068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/minicluster-data/ts-1/data/instance:
uuid: "56637e94a06047d9a4ee50a0d26932ca"
format_stamp: "Formatted at 2026-09-26 16:31:09 on dist-test-slave-khd3"
server_key: "17afec6a251203ce3ab42ac02e0528d8"
server_key_iv: "2b959dc8068867327719e3caad076b17"
server_key_version: "encryptionkey@0"
I20260926 16:31:09.187074 10068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/minicluster-data/ts-1/wal/instance:
uuid: "56637e94a06047d9a4ee50a0d26932ca"
format_stamp: "Formatted at 2026-09-26 16:31:09 on dist-test-slave-khd3"
server_key: "17afec6a251203ce3ab42ac02e0528d8"
server_key_iv: "2b959dc8068867327719e3caad076b17"
server_key_version: "encryptionkey@0"
I20260926 16:31:09.192457 10068 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 16:31:09.196233 10083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:09.197781 10068 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:31:09.197978 10068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/minicluster-data/ts-1/wal
uuid: "56637e94a06047d9a4ee50a0d26932ca"
format_stamp: "Formatted at 2026-09-26 16:31:09 on dist-test-slave-khd3"
server_key: "17afec6a251203ce3ab42ac02e0528d8"
server_key_iv: "2b959dc8068867327719e3caad076b17"
server_key_version: "encryptionkey@0"
I20260926 16:31:09.198205 10068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:31:09.219758 10068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:31:09.224370 10068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:31:09.224828 10068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:31:09.225800 10068 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:31:09.227169 10068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:31:09.227272 10068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:09.227419 10068 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:31:09.227494 10068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:09.260627 10068 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.2:38941
I20260926 16:31:09.260711 10195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.2:38941 every 8 connection(s)
I20260926 16:31:09.261933 10068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
I20260926 16:31:09.270533 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 10068
I20260926 16:31:09.270668 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/minicluster-data/ts-1/wal/instance
I20260926 16:31:09.271095 6276 external_mini_cluster.cc:1428] Setting key 3d85c6400f3829e4109e00ea042f02f2
Sep 26 16:31:09 dist-test-slave-khd3 krb5kdc[9828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440269, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.2@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:31:09.282424 10064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:09.264385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:43389 (local address 127.6.33.62:41009)
0926 16:31:09.264555 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:09.264559 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:09.265765 (+ 1206us) server_negotiation.cc:407] Connection header received
0926 16:31:09.267445 (+ 1680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:09.267449 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:09.267514 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:09.267603 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:09.268992 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:09.269673 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:09.271396 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:09.271971 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:09.272115 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:09.276007 (+ 3892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:09.276013 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:09.276018 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:09.276062 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:09.279251 (+ 3189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:09.279974 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:09.279980 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:09.279983 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:09.280124 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:09.280542 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:09.280547 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:09.280551 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:09.280892 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:09.281037 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:09.281856 (+ 819us) server_negotiation.cc:299] Negotiation successful
0926 16:31:09.282083 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 16:31:09.283419 10198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:09.264936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:41009 (local address 127.6.33.2:43389)
0926 16:31:09.265611 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:09.265647 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:09.267210 (+ 1563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:09.267795 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:09.267803 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:09.268295 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:09.268851 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:09.268868 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:09.269806 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:09.269809 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:09.271263 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:09.271275 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:09.272196 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:09.272203 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:09.272427 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:09.273182 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:09.273207 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:09.275787 (+ 2580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:31:09.279388 (+ 3601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:09.279397 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:09.279414 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:09.279840 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:09.280236 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:09.280242 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:09.280246 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:09.280427 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:09.281010 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:31:09.281018 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:31:09.281464 (+ 446us) client_negotiation.cc:769] Sending connection context
0926 16:31:09.281762 (+ 298us) client_negotiation.cc:240] Negotiation successful
0926 16:31:09.282299 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":254,"threads_started":1}
I20260926 16:31:09.284997 10196 heartbeater.cc:347] Connected to a master server at 127.6.33.62:41009
I20260926 16:31:09.285295 10196 heartbeater.cc:464] Registering TS with master...
I20260926 16:31:09.286008 10196 heartbeater.cc:511] Master 127.6.33.62:41009 requested a full tablet report, sending...
I20260926 16:31:09.287596 9873 ts_manager.cc:194] Registered new tserver with Master: 56637e94a06047d9a4ee50a0d26932ca (127.6.33.2:38941)
I20260926 16:31:09.288795 9873 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.33.2@KRBTEST.COM'} at 127.6.33.2:43389
WARNING: no policy specified for kudu/127.6.33.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab.
Entry for principal kudu/127.6.33.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab.
I20260926 16:31:09.316283 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.3:0
--local_ip_for_outbound_sockets=127.6.33.3
--webserver_interface=127.6.33.3
--webserver_port=0
--tserver_master_addrs=127.6.33.62:41009
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab
--principal=kudu/127.6.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:38175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:31:09.638744 10201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:31:09.639008 10201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:31:09.639173 10201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:31:09.646416 10201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:31:09.646529 10201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:31:09.646708 10201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.3
I20260926 16:31:09.656374 10201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:38175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab
--principal=kudu/127.6.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.33.3
--webserver_port=0
--tserver_master_addrs=127.6.33.62:41009
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.3
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:31:09.657815 10201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:31:09.659207 10201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:31:09.672376 10207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:09.672420 10206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:09.672453 10209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:31:09.673355 10201 server_base.cc:1034] running on GCE node
I20260926 16:31:09.674023 10201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:31:09.675041 10201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:31:09.676299 10201 hybrid_clock.cc:648] HybridClock initialized: now 1790440269676249 us; error 44 us; skew 500 ppm
I20260926 16:31:09.676687 10201 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:31:09 dist-test-slave-khd3 krb5kdc[9828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440269, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:31:09.680259 10201 init.cc:377] Logged in from keytab as kudu/127.6.33.3@KRBTEST.COM (short username kudu)
I20260926 16:31:09.681768 10201 webserver.cc:460] Webserver started at http://127.6.33.3:40721/ using document root <none> and password file <none>
I20260926 16:31:09.682456 10201 fs_manager.cc:359] Metadata directory not provided
I20260926 16:31:09.682621 10201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:31:09.682929 10201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:31:09.685997 10201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/minicluster-data/ts-2/data/instance:
uuid: "6f38bf6aaaa74acebbdefd3c8bc04509"
format_stamp: "Formatted at 2026-09-26 16:31:09 on dist-test-slave-khd3"
server_key: "def4d199782aa88d9a1be7611b74ee1f"
server_key_iv: "fb847b0ac535db3b3dba184ebb9f0e55"
server_key_version: "encryptionkey@0"
I20260926 16:31:09.686818 10201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/minicluster-data/ts-2/wal/instance:
uuid: "6f38bf6aaaa74acebbdefd3c8bc04509"
format_stamp: "Formatted at 2026-09-26 16:31:09 on dist-test-slave-khd3"
server_key: "def4d199782aa88d9a1be7611b74ee1f"
server_key_iv: "fb847b0ac535db3b3dba184ebb9f0e55"
server_key_version: "encryptionkey@0"
I20260926 16:31:09.692381 10201 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260926 16:31:09.696280 10216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:09.697945 10201 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:31:09.698119 10201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/minicluster-data/ts-2/wal
uuid: "6f38bf6aaaa74acebbdefd3c8bc04509"
format_stamp: "Formatted at 2026-09-26 16:31:09 on dist-test-slave-khd3"
server_key: "def4d199782aa88d9a1be7611b74ee1f"
server_key_iv: "fb847b0ac535db3b3dba184ebb9f0e55"
server_key_version: "encryptionkey@0"
I20260926 16:31:09.698318 10201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:31:09.727164 10201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:31:09.731629 10201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:31:09.731891 10201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:31:09.733039 10201 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:31:09.734417 10201 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:31:09.734490 10201 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:09.734580 10201 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:31:09.734623 10201 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:09.768203 10201 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.3:39517
I20260926 16:31:09.768251 10328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.3:39517 every 8 connection(s)
I20260926 16:31:09.769595 10201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
I20260926 16:31:09.775031 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 10201
I20260926 16:31:09.775226 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440233850748-6276-0/minicluster-data/ts-2/wal/instance
I20260926 16:31:09.775755 6276 external_mini_cluster.cc:1428] Setting key f4defbb3520082a7b031cd4b315ec435
Sep 26 16:31:09 dist-test-slave-khd3 krb5kdc[9828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440269, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.3@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:31:09.790623 10064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:09.772241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.3:38081 (local address 127.6.33.62:41009)
0926 16:31:09.772450 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:09.772456 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:09.773693 (+ 1237us) server_negotiation.cc:407] Connection header received
0926 16:31:09.776027 (+ 2334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:09.776032 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:09.776097 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:09.776182 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:09.778175 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:09.779259 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:09.780840 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:09.781212 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:09.781308 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:09.784842 (+ 3534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:09.784847 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:09.784850 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:09.784884 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:09.787451 (+ 2567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:09.788164 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:09.788167 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:09.788169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:09.788276 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:09.788700 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:09.788703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:09.788705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:09.789009 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:09.789143 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:09.790011 (+ 868us) server_negotiation.cc:299] Negotiation successful
0926 16:31:09.790262 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 16:31:09.791190 10062 heartbeater.cc:503] Master 127.6.33.62:41009 was elected leader, sending a full tablet report...
I20260926 16:31:09.791872 10331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:09.772846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:41009 (local address 127.6.33.3:38081)
0926 16:31:09.773518 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:09.773563 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:09.775694 (+ 2131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:09.776443 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:09.776451 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:09.777217 (+ 766us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:09.778014 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:09.778034 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:09.779388 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:09.779392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:09.780663 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:09.780675 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:09.781400 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:09.781405 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:09.781634 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:09.782336 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:09.782358 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:09.784557 (+ 2199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:31:09.787592 (+ 3035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:09.787599 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:09.787613 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:09.788048 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:09.788374 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:09.788378 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:09.788380 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:09.788609 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:09.789186 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:31:09.789193 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:31:09.789621 (+ 428us) client_negotiation.cc:769] Sending connection context
0926 16:31:09.789921 (+ 300us) client_negotiation.cc:240] Negotiation successful
0926 16:31:09.790452 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":267,"threads_started":1}
I20260926 16:31:09.793177 10329 heartbeater.cc:347] Connected to a master server at 127.6.33.62:41009
I20260926 16:31:09.793469 10329 heartbeater.cc:464] Registering TS with master...
I20260926 16:31:09.794121 10329 heartbeater.cc:511] Master 127.6.33.62:41009 requested a full tablet report, sending...
I20260926 16:31:09.795535 9873 ts_manager.cc:194] Registered new tserver with Master: 6f38bf6aaaa74acebbdefd3c8bc04509 (127.6.33.3:39517)
I20260926 16:31:09.796310 9873 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.33.3@KRBTEST.COM'} at 127.6.33.3:38081
I20260926 16:31:09.803844 6276 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 16:31:09 dist-test-slave-khd3 krb5kdc[9828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 16:31:09 dist-test-slave-khd3 krb5kdc[9828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.1@KRBTEST.COM
I20260926 16:31:09.852342 10336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:09.837200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33190 (local address 127.6.33.1:46107)
0926 16:31:09.837892 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:09.837899 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:09.837924 (+ 25us) server_negotiation.cc:407] Connection header received
0926 16:31:09.837983 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:09.837990 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:09.838211 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:09.838385 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:09.839863 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:09.840923 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:09.842217 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:09.842816 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:09.842953 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:09.846999 (+ 4046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:09.847011 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:09.847026 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:09.847076 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:09.849954 (+ 2878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:09.850586 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:09.850593 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:09.850596 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:09.850721 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:09.851059 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:09.851063 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:09.851065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:09.851400 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:09.851647 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:09.851859 (+ 212us) server_negotiation.cc:299] Negotiation successful
0926 16:31:09.852058 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":375,"threads_started":1}
Sep 26 16:31:09 dist-test-slave-khd3 krb5kdc[9828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:31:09.870472 10064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:09.856566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58296 (local address 127.6.33.62:41009)
0926 16:31:09.856810 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:09.856816 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:09.857004 (+ 188us) server_negotiation.cc:407] Connection header received
0926 16:31:09.857190 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:09.857196 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:09.857270 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:09.857382 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:09.858507 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:09.859225 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:09.860713 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:09.861150 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:09.861280 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:09.865053 (+ 3773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:09.865059 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:09.865062 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:09.865095 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:09.867805 (+ 2710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:09.868452 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:09.868460 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:09.868465 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:09.868589 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:09.869012 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:09.869017 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:09.869021 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:09.869331 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:09.869581 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:09.869915 (+ 334us) server_negotiation.cc:299] Negotiation successful
0926 16:31:09.870149 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260926 16:31:09.871038 9873 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58296
I20260926 16:31:09.872071 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 9934
I20260926 16:31:09.884953 10336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:09.873217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33200 (local address 127.6.33.1:46107)
0926 16:31:09.873522 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:09.873529 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:09.873698 (+ 169us) server_negotiation.cc:407] Connection header received
0926 16:31:09.873795 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:09.873798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:09.873885 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:09.873963 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:09.875106 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:09.875774 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:09.877040 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:09.877428 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:09.877520 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:09.879714 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:09.879718 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:09.879722 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:09.879751 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:09.882252 (+ 2501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:09.882836 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:09.882840 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:09.882842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:09.882956 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:09.883348 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:09.883352 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:09.883354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:09.883672 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:09.883853 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:09.884212 (+ 359us) server_negotiation.cc:299] Negotiation successful
0926 16:31:09.884381 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 16:31:09.885530 10057 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:31:09.969624 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 9934
I20260926 16:31:09.986696 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 10068
Sep 26 16:31:09 dist-test-slave-khd3 krb5kdc[9828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.2@KRBTEST.COM
I20260926 16:31:10.002908 10341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:09.987620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55276 (local address 127.6.33.2:38941)
0926 16:31:09.988161 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:09.988168 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:09.988192 (+ 24us) server_negotiation.cc:407] Connection header received
0926 16:31:09.988249 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:09.988254 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:09.988453 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:09.988641 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:09.989870 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:09.990890 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:09.992204 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:09.992853 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:09.992990 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:09.997100 (+ 4110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:09.997113 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:09.997128 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:09.997173 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:09.999945 (+ 2772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:10.000680 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:10.000687 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:10.000690 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:10.000885 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:10.001269 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:10.001275 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:10.001279 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:10.001684 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:10.001950 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:10.002294 (+ 344us) server_negotiation.cc:299] Negotiation successful
0926 16:31:10.002500 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":264,"threads_started":1}
I20260926 16:31:10.004216 10191 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:31:10.087769 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 10068
I20260926 16:31:10.104249 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 10201
Sep 26 16:31:10 dist-test-slave-khd3 krb5kdc[9828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.3@KRBTEST.COM
I20260926 16:31:10.121426 10343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:10.105311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59764 (local address 127.6.33.3:39517)
0926 16:31:10.106045 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:10.106052 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:10.106082 (+ 30us) server_negotiation.cc:407] Connection header received
0926 16:31:10.106140 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:10.106146 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:10.106362 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:10.106494 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:10.107920 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:10.109420 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:10.110686 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:10.111268 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:10.111373 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:10.115680 (+ 4307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:10.115693 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:10.115708 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:10.115744 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:10.118896 (+ 3152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:10.119553 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:10.119560 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:10.119563 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:10.119689 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:10.119979 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:10.119982 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:10.119984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:10.120305 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:10.120589 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:10.120833 (+ 244us) server_negotiation.cc:299] Negotiation successful
0926 16:31:10.121028 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":404,"threads_started":1}
I20260926 16:31:10.122787 10324 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:31:10.208187 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 10201
I20260926 16:31:10.224083 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 9841
I20260926 16:31:10.225286 9903 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:31:10.327637 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 9841
2026-09-26T16:31:10Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3893 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:31:10 dist-test-slave-khd3 krb5kdc[10348](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:31:10 dist-test-slave-khd3 krb5kdc[10348](info): set up 2 sockets
Sep 26 16:31:10 dist-test-slave-khd3 krb5kdc[10348](info): commencing operation
krb5kdc: starting...
W20260926 16:31:11.414191 6276 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Sep 26 16:31:11 dist-test-slave-khd3 krb5kdc[10348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:31:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:31:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab.
Entry for principal kudu/127.6.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab.
I20260926 16:31:11.612448 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:42383
--webserver_interface=127.6.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:46835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.33.62:42383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:31:11.932188 10361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:31:11.932464 10361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:31:11.932559 10361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:31:11.939850 10361 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:31:11.939960 10361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:31:11.940006 10361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:31:11.940053 10361 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:31:11.940102 10361 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:31:11.949976 10361 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:46835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.33.62:42383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:42383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:31:11.951216 10361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:31:11.952502 10361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:31:11.965667 10367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:11.965771 10369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:11.965799 10366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:31:11.966629 10361 server_base.cc:1034] running on GCE node
I20260926 16:31:11.967378 10361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:31:11.968966 10361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:31:11.970208 10361 hybrid_clock.cc:648] HybridClock initialized: now 1790440271970160 us; error 42 us; skew 500 ppm
I20260926 16:31:11.970582 10361 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:31:11 dist-test-slave-khd3 krb5kdc[10348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440271, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:31:11.979887 10361 init.cc:377] Logged in from keytab as kudu/127.6.33.62@KRBTEST.COM (short username kudu)
I20260926 16:31:11.981209 10361 fs_manager.cc:359] Metadata directory not provided
I20260926 16:31:11.981345 10361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:31:11.981710 10361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:31:11.985971 10361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/master-0/data/instance:
uuid: "1f17986b5f1a4c079f1b74a9a7953b85"
format_stamp: "Formatted at 2026-09-26 16:31:11 on dist-test-slave-khd3"
server_key: "41b28e6efe1fd12bfb73f8e102602b1f"
server_key_iv: "1c041be95bd11519a488d2841c5211bb"
server_key_version: "encryptionkey@0"
I20260926 16:31:11.987166 10361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/master-0/wal/instance:
uuid: "1f17986b5f1a4c079f1b74a9a7953b85"
format_stamp: "Formatted at 2026-09-26 16:31:11 on dist-test-slave-khd3"
server_key: "41b28e6efe1fd12bfb73f8e102602b1f"
server_key_iv: "1c041be95bd11519a488d2841c5211bb"
server_key_version: "encryptionkey@0"
I20260926 16:31:11.994616 10361 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260926 16:31:11.999670 10375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:12.001488 10361 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 16:31:12.001698 10361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/master-0/wal
uuid: "1f17986b5f1a4c079f1b74a9a7953b85"
format_stamp: "Formatted at 2026-09-26 16:31:11 on dist-test-slave-khd3"
server_key: "41b28e6efe1fd12bfb73f8e102602b1f"
server_key_iv: "1c041be95bd11519a488d2841c5211bb"
server_key_version: "encryptionkey@0"
I20260926 16:31:12.001987 10361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:31:12.022486 10361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:31:12.028371 10361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:31:12.028676 10361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:31:12.047362 10361 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.62:42383
I20260926 16:31:12.047382 10426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.62:42383 every 8 connection(s)
I20260926 16:31:12.048517 10361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
I20260926 16:31:12.051702 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 10361
I20260926 16:31:12.051925 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/master-0/wal/instance
I20260926 16:31:12.052718 6276 external_mini_cluster.cc:1428] Setting key 6b98a444d435fb01d159d2cb284a0135
I20260926 16:31:12.052975 10427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:31:12.066229 10427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f17986b5f1a4c079f1b74a9a7953b85: Bootstrap starting.
Sep 26 16:31:12 dist-test-slave-khd3 krb5kdc[10348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:31:12.069854 10427 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f17986b5f1a4c079f1b74a9a7953b85: Neither blocks nor log segments found. Creating new log.
I20260926 16:31:12.070987 10427 log.cc:824] T 00000000000000000000000000000000 P 1f17986b5f1a4c079f1b74a9a7953b85: Log is configured to *not* fsync() on all Append() calls
I20260926 16:31:12.074035 10427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f17986b5f1a4c079f1b74a9a7953b85: No bootstrap required, opened a new log
I20260926 16:31:12.076212 10430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:12.054977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40110 (local address 127.6.33.62:42383)
0926 16:31:12.055837 (+ 860us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:12.055849 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:12.055895 (+ 46us) server_negotiation.cc:407] Connection header received
0926 16:31:12.057954 (+ 2059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:12.057996 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:12.058469 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:12.058955 (+ 486us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:12.060556 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:12.061994 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:12.063322 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:12.064103 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:12.064312 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:12.068710 (+ 4398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:12.068719 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:12.068737 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:12.068849 (+ 112us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:12.072162 (+ 3313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:12.072905 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:12.072921 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:12.072927 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:12.073097 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:12.073483 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:12.073487 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:12.073493 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:12.074014 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:12.074319 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:12.074795 (+ 476us) server_negotiation.cc:299] Negotiation successful
0926 16:31:12.075197 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":382,"threads_started":1}
I20260926 16:31:12.082078 10427 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f17986b5f1a4c079f1b74a9a7953b85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f17986b5f1a4c079f1b74a9a7953b85" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 42383 } }
I20260926 16:31:12.082461 10427 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f17986b5f1a4c079f1b74a9a7953b85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:31:12.082594 10427 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f17986b5f1a4c079f1b74a9a7953b85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f17986b5f1a4c079f1b74a9a7953b85, State: Initialized, Role: FOLLOWER
I20260926 16:31:12.083148 10427 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f17986b5f1a4c079f1b74a9a7953b85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f17986b5f1a4c079f1b74a9a7953b85" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 42383 } }
I20260926 16:31:12.083382 10427 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f17986b5f1a4c079f1b74a9a7953b85 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:31:12.083529 10427 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f17986b5f1a4c079f1b74a9a7953b85 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:31:12.083709 10427 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f17986b5f1a4c079f1b74a9a7953b85 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:31:12.085935 10427 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f17986b5f1a4c079f1b74a9a7953b85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f17986b5f1a4c079f1b74a9a7953b85" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 42383 } }
I20260926 16:31:12.086423 10427 leader_election.cc:302] T 00000000000000000000000000000000 P 1f17986b5f1a4c079f1b74a9a7953b85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f17986b5f1a4c079f1b74a9a7953b85; no voters:
I20260926 16:31:12.087070 10427 leader_election.cc:288] T 00000000000000000000000000000000 P 1f17986b5f1a4c079f1b74a9a7953b85 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:31:12.087234 10432 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f17986b5f1a4c079f1b74a9a7953b85 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:31:12.088097 10432 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f17986b5f1a4c079f1b74a9a7953b85 [term 1 LEADER]: Becoming Leader. State: Replica: 1f17986b5f1a4c079f1b74a9a7953b85, State: Running, Role: LEADER
I20260926 16:31:12.088804 10432 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f17986b5f1a4c079f1b74a9a7953b85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f17986b5f1a4c079f1b74a9a7953b85" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 42383 } }
I20260926 16:31:12.089570 10427 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f17986b5f1a4c079f1b74a9a7953b85 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:31:12.092042 10433 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f17986b5f1a4c079f1b74a9a7953b85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f17986b5f1a4c079f1b74a9a7953b85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f17986b5f1a4c079f1b74a9a7953b85" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 42383 } } }
I20260926 16:31:12.092406 10433 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f17986b5f1a4c079f1b74a9a7953b85 [sys.catalog]: This master's current role is: LEADER
I20260926 16:31:12.093201 10434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f17986b5f1a4c079f1b74a9a7953b85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f17986b5f1a4c079f1b74a9a7953b85. Latest consensus state: current_term: 1 leader_uuid: "1f17986b5f1a4c079f1b74a9a7953b85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f17986b5f1a4c079f1b74a9a7953b85" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 42383 } } }
I20260926 16:31:12.093592 10434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f17986b5f1a4c079f1b74a9a7953b85 [sys.catalog]: This master's current role is: LEADER
I20260926 16:31:12.094122 10438 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:31:12.100014 10438 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:31:12.110546 10438 catalog_manager.cc:1347] Generated new cluster ID: 676303ac7cde4e3093d3e99516194d99
I20260926 16:31:12.110678 10438 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:31:12.118104 10438 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:31:12.120152 10438 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:31:12.126910 10438 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f17986b5f1a4c079f1b74a9a7953b85: Generated new TSK 0
I20260926 16:31:12.127732 10438 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.33.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab.
Entry for principal kudu/127.6.33.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab.
I20260926 16:31:12.182337 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.1:0
--local_ip_for_outbound_sockets=127.6.33.1
--webserver_interface=127.6.33.1
--webserver_port=0
--tserver_master_addrs=127.6.33.62:42383
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab
--principal=kudu/127.6.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:46835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:31:12.513749 10453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:31:12.514075 10453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:31:12.514242 10453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:31:12.521493 10453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:31:12.521667 10453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:31:12.521850 10453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.1
I20260926 16:31:12.531906 10453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:46835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab
--principal=kudu/127.6.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.33.1
--webserver_port=0
--tserver_master_addrs=127.6.33.62:42383
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.1
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:31:12.533277 10453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:31:12.534550 10453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:31:12.547896 10458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:12.547996 10459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:12.547878 10461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:31:12.548878 10453 server_base.cc:1034] running on GCE node
I20260926 16:31:12.549448 10453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:31:12.550400 10453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:31:12.551666 10453 hybrid_clock.cc:648] HybridClock initialized: now 1790440272551595 us; error 78 us; skew 500 ppm
I20260926 16:31:12.552032 10453 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:31:12 dist-test-slave-khd3 krb5kdc[10348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440272, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:31:12.555380 10453 init.cc:377] Logged in from keytab as kudu/127.6.33.1@KRBTEST.COM (short username kudu)
I20260926 16:31:12.556655 10453 fs_manager.cc:359] Metadata directory not provided
I20260926 16:31:12.556825 10453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:31:12.557191 10453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:31:12.560120 10453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/ts-0/data/instance:
uuid: "39b6750474b0461f98af5f4b3dd3444b"
format_stamp: "Formatted at 2026-09-26 16:31:12 on dist-test-slave-khd3"
server_key: "c8b2a24ae5bc441b5bd15a95606dc2c2"
server_key_iv: "068657f9d98dc00f51cc39ed91907a7f"
server_key_version: "encryptionkey@0"
I20260926 16:31:12.560998 10453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/ts-0/wal/instance:
uuid: "39b6750474b0461f98af5f4b3dd3444b"
format_stamp: "Formatted at 2026-09-26 16:31:12 on dist-test-slave-khd3"
server_key: "c8b2a24ae5bc441b5bd15a95606dc2c2"
server_key_iv: "068657f9d98dc00f51cc39ed91907a7f"
server_key_version: "encryptionkey@0"
I20260926 16:31:12.566130 10453 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 16:31:12.570215 10467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:12.571754 10453 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:31:12.571969 10453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/ts-0/wal
uuid: "39b6750474b0461f98af5f4b3dd3444b"
format_stamp: "Formatted at 2026-09-26 16:31:12 on dist-test-slave-khd3"
server_key: "c8b2a24ae5bc441b5bd15a95606dc2c2"
server_key_iv: "068657f9d98dc00f51cc39ed91907a7f"
server_key_version: "encryptionkey@0"
I20260926 16:31:12.572207 10453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:31:12.599303 10453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:31:12.603802 10453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:31:12.604100 10453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:31:12.605098 10453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:31:12.606426 10453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:31:12.606552 10453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:12.606685 10453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:31:12.606772 10453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:12.640627 10453 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.1:41389
I20260926 16:31:12.640719 10579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.1:41389 every 8 connection(s)
I20260926 16:31:12.641525 10453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
I20260926 16:31:12.644527 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 10453
I20260926 16:31:12.644795 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/ts-0/wal/instance
I20260926 16:31:12.645684 6276 external_mini_cluster.cc:1428] Setting key e2988860cf966e3171fb70bf4a47e8e8
Sep 26 16:31:12 dist-test-slave-khd3 krb5kdc[10348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440272, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.1@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:31:12.665668 10582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:12.644016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:60193 (local address 127.6.33.62:42383)
0926 16:31:12.644533 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:12.644540 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:12.645765 (+ 1225us) server_negotiation.cc:407] Connection header received
0926 16:31:12.647967 (+ 2202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:12.647971 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:12.648053 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:12.648140 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:12.650355 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:12.651072 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:12.653689 (+ 2617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:12.654476 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:12.654642 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:12.658976 (+ 4334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:12.658981 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:12.658985 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:12.659020 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:12.661767 (+ 2747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:12.662902 (+ 1135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:12.662907 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:12.662909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:12.663045 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:12.663670 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:12.663675 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:12.663677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:12.663963 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:12.664108 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:12.665174 (+ 1066us) server_negotiation.cc:299] Negotiation successful
0926 16:31:12.665359 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":242,"threads_started":1}
WARNING: no policy specified for kudu/127.6.33.2@KRBTEST.COM; defaulting to no policy
I20260926 16:31:12.667284 10583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:12.644628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:42383 (local address 127.6.33.1:60193)
0926 16:31:12.645566 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:12.645625 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:12.647657 (+ 2032us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:12.648427 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:12.648440 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:12.649276 (+ 836us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:12.650151 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:12.650180 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:12.651246 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:12.651256 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:12.653487 (+ 2231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:12.653507 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:12.654753 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:12.654761 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:12.655124 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:12.655925 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:12.655950 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:12.658672 (+ 2722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:31:12.661968 (+ 3296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:12.661981 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:12.662008 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:12.662731 (+ 723us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:12.663176 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:12.663183 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:12.663189 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:12.663497 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:12.664110 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:31:12.664119 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:31:12.664631 (+ 512us) client_negotiation.cc:769] Sending connection context
0926 16:31:12.665028 (+ 397us) client_negotiation.cc:240] Negotiation successful
0926 16:31:12.665645 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":371,"threads_started":1}
I20260926 16:31:12.669147 10580 heartbeater.cc:347] Connected to a master server at 127.6.33.62:42383
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.2@KRBTEST.COM" created.
I20260926 16:31:12.669580 10580 heartbeater.cc:464] Registering TS with master...
I20260926 16:31:12.670318 10580 heartbeater.cc:511] Master 127.6.33.62:42383 requested a full tablet report, sending...
I20260926 16:31:12.672657 10392 ts_manager.cc:194] Registered new tserver with Master: 39b6750474b0461f98af5f4b3dd3444b (127.6.33.1:41389)
I20260926 16:31:12.674578 10392 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.33.1@KRBTEST.COM'} at 127.6.33.1:60193
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab.
Entry for principal kudu/127.6.33.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab.
I20260926 16:31:12.696026 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.2:0
--local_ip_for_outbound_sockets=127.6.33.2
--webserver_interface=127.6.33.2
--webserver_port=0
--tserver_master_addrs=127.6.33.62:42383
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab
--principal=kudu/127.6.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:46835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:31:13.027590 10586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:31:13.027892 10586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:31:13.028002 10586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:31:13.035576 10586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:31:13.035697 10586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:31:13.035815 10586 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.2
I20260926 16:31:13.045751 10586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:46835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab
--principal=kudu/127.6.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.33.2
--webserver_port=0
--tserver_master_addrs=127.6.33.62:42383
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.2
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:31:13.047079 10586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:31:13.048429 10586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:31:13.063864 10591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:13.063898 10592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:13.064167 10594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:31:13.065487 10586 server_base.cc:1034] running on GCE node
I20260926 16:31:13.066145 10586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:31:13.067188 10586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:31:13.068391 10586 hybrid_clock.cc:648] HybridClock initialized: now 1790440273068346 us; error 37 us; skew 500 ppm
I20260926 16:31:13.068737 10586 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:31:13 dist-test-slave-khd3 krb5kdc[10348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440273, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:31:13.072005 10586 init.cc:377] Logged in from keytab as kudu/127.6.33.2@KRBTEST.COM (short username kudu)
I20260926 16:31:13.073365 10586 fs_manager.cc:359] Metadata directory not provided
I20260926 16:31:13.073455 10586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:31:13.073719 10586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:31:13.076923 10586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/ts-1/data/instance:
uuid: "460d8bf451e44c29a2e3066f83241f65"
format_stamp: "Formatted at 2026-09-26 16:31:13 on dist-test-slave-khd3"
server_key: "dadc18e62d4a86af5a8e1682f4f44176"
server_key_iv: "526150661ef43887476602cd94d2899d"
server_key_version: "encryptionkey@0"
I20260926 16:31:13.077808 10586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/ts-1/wal/instance:
uuid: "460d8bf451e44c29a2e3066f83241f65"
format_stamp: "Formatted at 2026-09-26 16:31:13 on dist-test-slave-khd3"
server_key: "dadc18e62d4a86af5a8e1682f4f44176"
server_key_iv: "526150661ef43887476602cd94d2899d"
server_key_version: "encryptionkey@0"
I20260926 16:31:13.082844 10586 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 16:31:13.086966 10600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:13.088372 10586 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 16:31:13.088552 10586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/ts-1/wal
uuid: "460d8bf451e44c29a2e3066f83241f65"
format_stamp: "Formatted at 2026-09-26 16:31:13 on dist-test-slave-khd3"
server_key: "dadc18e62d4a86af5a8e1682f4f44176"
server_key_iv: "526150661ef43887476602cd94d2899d"
server_key_version: "encryptionkey@0"
I20260926 16:31:13.088835 10586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:31:13.102644 10586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:31:13.107105 10586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:31:13.107403 10586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:31:13.108493 10586 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:31:13.109916 10586 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:31:13.110061 10586 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:13.110193 10586 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:31:13.110286 10586 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:13.145282 10586 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.2:39703
I20260926 16:31:13.145327 10712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.2:39703 every 8 connection(s)
I20260926 16:31:13.146250 10586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
I20260926 16:31:13.156260 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 10586
I20260926 16:31:13.156492 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/ts-1/wal/instance
I20260926 16:31:13.157024 6276 external_mini_cluster.cc:1428] Setting key f0f632cc0760ac8570a43ca8dede6b5c
Sep 26 16:31:13 dist-test-slave-khd3 krb5kdc[10348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440273, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.2@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:31:13.167574 10582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:13.149016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:40529 (local address 127.6.33.62:42383)
0926 16:31:13.149209 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:13.149214 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:13.150493 (+ 1279us) server_negotiation.cc:407] Connection header received
0926 16:31:13.152398 (+ 1905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:13.152404 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:13.152502 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:13.152617 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:13.154288 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:13.154995 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:13.156704 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:13.157255 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:13.157395 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:13.161286 (+ 3891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:13.161291 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:13.161295 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:13.161327 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:13.164175 (+ 2848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:13.165152 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:13.165163 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:13.165165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:13.165294 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:13.165756 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:13.165764 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:13.165771 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:13.166059 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:13.166216 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:13.166957 (+ 741us) server_negotiation.cc:299] Negotiation successful
0926 16:31:13.167224 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 16:31:13.168723 10715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:13.149534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:42383 (local address 127.6.33.2:40529)
0926 16:31:13.150324 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:13.150363 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:13.152101 (+ 1738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:13.152941 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:13.152951 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:13.153513 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:13.154100 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:13.154117 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:13.155146 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:13.155152 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:13.156529 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:13.156563 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:13.157515 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:13.157524 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:13.157765 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:13.158540 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:13.158576 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:13.161027 (+ 2451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:31:13.164382 (+ 3355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:13.164393 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:13.164418 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:13.165009 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:13.165419 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:13.165424 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:13.165428 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:13.165627 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:13.166202 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:31:13.166210 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:31:13.166582 (+ 372us) client_negotiation.cc:769] Sending connection context
0926 16:31:13.166861 (+ 279us) client_negotiation.cc:240] Negotiation successful
0926 16:31:13.167447 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":340,"threads_started":1}
I20260926 16:31:13.170534 10713 heartbeater.cc:347] Connected to a master server at 127.6.33.62:42383
I20260926 16:31:13.170877 10713 heartbeater.cc:464] Registering TS with master...
I20260926 16:31:13.171604 10713 heartbeater.cc:511] Master 127.6.33.62:42383 requested a full tablet report, sending...
I20260926 16:31:13.173204 10392 ts_manager.cc:194] Registered new tserver with Master: 460d8bf451e44c29a2e3066f83241f65 (127.6.33.2:39703)
I20260926 16:31:13.173995 10392 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.33.2@KRBTEST.COM'} at 127.6.33.2:40529
WARNING: no policy specified for kudu/127.6.33.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab.
Entry for principal kudu/127.6.33.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab.
I20260926 16:31:13.206244 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.3:0
--local_ip_for_outbound_sockets=127.6.33.3
--webserver_interface=127.6.33.3
--webserver_port=0
--tserver_master_addrs=127.6.33.62:42383
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab
--principal=kudu/127.6.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:46835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:31:13.531414 10718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:31:13.531760 10718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:31:13.531859 10718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:31:13.539057 10718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:31:13.539179 10718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:31:13.539297 10718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.3
I20260926 16:31:13.549081 10718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:46835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab
--principal=kudu/127.6.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.33.3
--webserver_port=0
--tserver_master_addrs=127.6.33.62:42383
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.3
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:31:13.550357 10718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:31:13.551632 10718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:31:13.564659 10723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:13.564730 10724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:13.564677 10726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:31:13.566320 10718 server_base.cc:1034] running on GCE node
I20260926 16:31:13.566892 10718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:31:13.567831 10718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:31:13.569098 10718 hybrid_clock.cc:648] HybridClock initialized: now 1790440273569060 us; error 56 us; skew 500 ppm
I20260926 16:31:13.569588 10718 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:31:13 dist-test-slave-khd3 krb5kdc[10348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440273, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:31:13.573026 10718 init.cc:377] Logged in from keytab as kudu/127.6.33.3@KRBTEST.COM (short username kudu)
I20260926 16:31:13.574208 10718 fs_manager.cc:359] Metadata directory not provided
I20260926 16:31:13.574299 10718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:31:13.574579 10718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:31:13.577997 10718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/ts-2/data/instance:
uuid: "9edb1699aa294f3f98bf4ef560ed1e86"
format_stamp: "Formatted at 2026-09-26 16:31:13 on dist-test-slave-khd3"
server_key: "8f5c35cbd6c8c4f063a5d99d07329b6d"
server_key_iv: "6bfa948269309b1d56ef9c519346de6b"
server_key_version: "encryptionkey@0"
I20260926 16:31:13.578842 10718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/ts-2/wal/instance:
uuid: "9edb1699aa294f3f98bf4ef560ed1e86"
format_stamp: "Formatted at 2026-09-26 16:31:13 on dist-test-slave-khd3"
server_key: "8f5c35cbd6c8c4f063a5d99d07329b6d"
server_key_iv: "6bfa948269309b1d56ef9c519346de6b"
server_key_version: "encryptionkey@0"
I20260926 16:31:13.584106 10718 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 16:31:13.587858 10732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:13.589406 10718 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 16:31:13.589599 10718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/ts-2/wal
uuid: "9edb1699aa294f3f98bf4ef560ed1e86"
format_stamp: "Formatted at 2026-09-26 16:31:13 on dist-test-slave-khd3"
server_key: "8f5c35cbd6c8c4f063a5d99d07329b6d"
server_key_iv: "6bfa948269309b1d56ef9c519346de6b"
server_key_version: "encryptionkey@0"
I20260926 16:31:13.589811 10718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:31:13.619761 10718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:31:13.624418 10718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:31:13.624794 10718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:31:13.626092 10718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:31:13.628249 10718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:31:13.628357 10718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:13.628465 10718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:31:13.628552 10718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:13.667064 10718 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.3:38689
I20260926 16:31:13.667111 10844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.3:38689 every 8 connection(s)
I20260926 16:31:13.667984 10718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
I20260926 16:31:13.676920 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 10718
I20260926 16:31:13.677095 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/ts-2/wal/instance
I20260926 16:31:13.677675 6276 external_mini_cluster.cc:1428] Setting key a5761fe1fce2eeda498ff3b72d18b147
I20260926 16:31:13.678417 10580 heartbeater.cc:503] Master 127.6.33.62:42383 was elected leader, sending a full tablet report...
Sep 26 16:31:13 dist-test-slave-khd3 krb5kdc[10348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440273, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.3@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:31:13.690167 10847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:13.670753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.3:54851 (local address 127.6.33.62:42383)
0926 16:31:13.671168 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:13.671173 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:13.672435 (+ 1262us) server_negotiation.cc:407] Connection header received
0926 16:31:13.674402 (+ 1967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:13.674407 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:13.674477 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:13.674587 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:13.676192 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:13.677509 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:13.679600 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:13.680265 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:13.680420 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:13.684026 (+ 3606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:13.684033 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:13.684038 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:13.684080 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:13.686766 (+ 2686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:13.687572 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:13.687581 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:13.687585 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:13.687746 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:13.688194 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:13.688199 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:13.688203 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:13.688491 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:13.688649 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:13.689522 (+ 873us) server_negotiation.cc:299] Negotiation successful
0926 16:31:13.689774 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":202,"threads_started":1}
I20260926 16:31:13.691493 10848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:13.671385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:42383 (local address 127.6.33.3:54851)
0926 16:31:13.672240 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:13.672299 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:13.674125 (+ 1826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:13.674806 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:13.674813 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:13.675339 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:13.676010 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:13.676027 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:13.677655 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:13.677659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:13.679455 (+ 1796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:13.679467 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:13.680510 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:13.680518 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:13.680832 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:13.681597 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:13.681625 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:13.683786 (+ 2161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:31:13.686964 (+ 3178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:13.686981 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:13.686994 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:13.687434 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:13.687865 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:13.687870 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:13.687874 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:13.688077 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:13.688624 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:31:13.688633 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:31:13.689067 (+ 434us) client_negotiation.cc:769] Sending connection context
0926 16:31:13.689424 (+ 357us) client_negotiation.cc:240] Negotiation successful
0926 16:31:13.690030 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":383,"threads_started":1}
I20260926 16:31:13.693038 10845 heartbeater.cc:347] Connected to a master server at 127.6.33.62:42383
I20260926 16:31:13.693414 10845 heartbeater.cc:464] Registering TS with master...
I20260926 16:31:13.694185 10845 heartbeater.cc:511] Master 127.6.33.62:42383 requested a full tablet report, sending...
I20260926 16:31:13.695859 10391 ts_manager.cc:194] Registered new tserver with Master: 9edb1699aa294f3f98bf4ef560ed1e86 (127.6.33.3:38689)
I20260926 16:31:13.697127 10391 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.33.3@KRBTEST.COM'} at 127.6.33.3:54851
I20260926 16:31:13.706218 6276 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:31:13.723228 10847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:13.710887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40112 (local address 127.6.33.62:42383)
0926 16:31:13.711085 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:13.711089 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:13.711563 (+ 474us) server_negotiation.cc:407] Connection header received
0926 16:31:13.711701 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:13.711705 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:13.711770 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:13.711856 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:13.713374 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:13.714106 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:13.715321 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:13.715815 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:13.715919 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:13.718358 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:13.718363 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:13.718366 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:13.718419 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:13.720931 (+ 2512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:13.721580 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:13.721583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:13.721586 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:13.721694 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:13.722001 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:13.722003 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:13.722005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:13.722310 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:13.722483 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:13.722821 (+ 338us) server_negotiation.cc:299] Negotiation successful
0926 16:31:13.722998 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260926 16:31:13.727433 10391 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40112:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:31:13.734328 10391 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:31:13.757010 10859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:13.746482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58776 (local address 127.6.33.2:39703)
0926 16:31:13.748705 (+ 2223us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:13.748712 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:13.748735 (+ 23us) server_negotiation.cc:407] Connection header received
0926 16:31:13.748875 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:13.748880 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:13.749067 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:13.750075 (+ 1008us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:31:13.750310 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:13.751893 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:13.754879 (+ 2986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:13.755797 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:13.755958 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:13.756174 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:13.756483 (+ 309us) server_negotiation.cc:299] Negotiation successful
0926 16:31:13.756656 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2062,"thread_start_us":1953,"threads_started":1}
I20260926 16:31:13.757085 10858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:13.746369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.2:39703 (local address 127.0.0.1:58776)
0926 16:31:13.747227 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:13.747272 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:13.747435 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:13.749334 (+ 1899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:13.749343 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:13.749375 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:31:13.749851 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:13.749864 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:13.752057 (+ 2193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:13.752068 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:13.754732 (+ 2664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:13.754744 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:13.756031 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:13.756047 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:13.756260 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:13.756305 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 16:31:13.756440 (+ 135us) client_negotiation.cc:240] Negotiation successful
0926 16:31:13.756636 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"mutex_wait_us":29,"thread_start_us":818,"threads_started":1}
I20260926 16:31:13.760257 10860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:13.746734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.3:38689 (local address 127.0.0.1:39784)
0926 16:31:13.748043 (+ 1309us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:13.748065 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:13.748195 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:13.751937 (+ 3742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:13.751942 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:13.751967 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:31:13.752673 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:13.752690 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:13.754574 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:13.754580 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:13.757975 (+ 3395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:13.757985 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:13.759347 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:13.759353 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:13.759520 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:13.759559 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 16:31:13.759675 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 16:31:13.759821 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1152,"mutex_wait_us":125,"thread_start_us":973,"threads_started":1}
I20260926 16:31:13.760555 10863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:13.748855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51644 (local address 127.6.33.1:41389)
0926 16:31:13.750668 (+ 1813us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:13.750677 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:13.751190 (+ 513us) server_negotiation.cc:407] Connection header received
0926 16:31:13.751477 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:13.751485 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:13.751706 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:13.751859 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:31:13.752976 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:13.754518 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:13.758132 (+ 3614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:13.759551 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:13.759677 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:13.759888 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:13.760106 (+ 218us) server_negotiation.cc:299] Negotiation successful
0926 16:31:13.760253 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1654,"thread_start_us":1501,"threads_started":1}
I20260926 16:31:13.760566 10862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:13.748193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:41389 (local address 127.0.0.1:51644)
0926 16:31:13.751093 (+ 2900us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:13.751110 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:13.751294 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:13.752142 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:13.752148 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:13.752186 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:31:13.752741 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:13.752751 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:13.754661 (+ 1910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:13.754669 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:13.757243 (+ 2574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:13.757261 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:13.759751 (+ 2490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:13.759757 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:13.759906 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:13.759946 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 16:31:13.760025 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 16:31:13.760145 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2769,"mutex_wait_us":180,"thread_start_us":2650,"threads_started":1}
I20260926 16:31:13.761561 10861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:13.747463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39784 (local address 127.6.33.3:38689)
0926 16:31:13.751489 (+ 4026us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:13.751495 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:13.751520 (+ 25us) server_negotiation.cc:407] Connection header received
0926 16:31:13.751598 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:13.751603 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:13.751761 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:13.751959 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:31:13.753188 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:13.754445 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:13.758123 (+ 3678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:13.759157 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:13.759302 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:13.759524 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:13.760994 (+ 1470us) server_negotiation.cc:299] Negotiation successful
0926 16:31:13.761177 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3885,"thread_start_us":3734,"threads_started":1}
I20260926 16:31:13.761210 10648 tablet_service.cc:1467] Processing CreateTablet for tablet 7b60b7c168a3455686ff11ed526ff3c2 (DEFAULT_TABLE table=test-table [id=8fd88584a04d4eb58e3212323c7daa3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:31:13.762730 10648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b60b7c168a3455686ff11ed526ff3c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:31:13.763907 10515 tablet_service.cc:1467] Processing CreateTablet for tablet 7b60b7c168a3455686ff11ed526ff3c2 (DEFAULT_TABLE table=test-table [id=8fd88584a04d4eb58e3212323c7daa3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:31:13.765195 10515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b60b7c168a3455686ff11ed526ff3c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:31:13.765365 10780 tablet_service.cc:1467] Processing CreateTablet for tablet 7b60b7c168a3455686ff11ed526ff3c2 (DEFAULT_TABLE table=test-table [id=8fd88584a04d4eb58e3212323c7daa3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:31:13.766659 10780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b60b7c168a3455686ff11ed526ff3c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:31:13.776068 10864 tablet_bootstrap.cc:492] T 7b60b7c168a3455686ff11ed526ff3c2 P 39b6750474b0461f98af5f4b3dd3444b: Bootstrap starting.
I20260926 16:31:13.779735 10864 tablet_bootstrap.cc:654] T 7b60b7c168a3455686ff11ed526ff3c2 P 39b6750474b0461f98af5f4b3dd3444b: Neither blocks nor log segments found. Creating new log.
I20260926 16:31:13.780906 10864 log.cc:824] T 7b60b7c168a3455686ff11ed526ff3c2 P 39b6750474b0461f98af5f4b3dd3444b: Log is configured to *not* fsync() on all Append() calls
I20260926 16:31:13.781739 10865 tablet_bootstrap.cc:492] T 7b60b7c168a3455686ff11ed526ff3c2 P 460d8bf451e44c29a2e3066f83241f65: Bootstrap starting.
I20260926 16:31:13.784914 10866 tablet_bootstrap.cc:492] T 7b60b7c168a3455686ff11ed526ff3c2 P 9edb1699aa294f3f98bf4ef560ed1e86: Bootstrap starting.
I20260926 16:31:13.785598 10864 tablet_bootstrap.cc:492] T 7b60b7c168a3455686ff11ed526ff3c2 P 39b6750474b0461f98af5f4b3dd3444b: No bootstrap required, opened a new log
I20260926 16:31:13.785754 10865 tablet_bootstrap.cc:654] T 7b60b7c168a3455686ff11ed526ff3c2 P 460d8bf451e44c29a2e3066f83241f65: Neither blocks nor log segments found. Creating new log.
I20260926 16:31:13.785881 10864 ts_tablet_manager.cc:1397] T 7b60b7c168a3455686ff11ed526ff3c2 P 39b6750474b0461f98af5f4b3dd3444b: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260926 16:31:13.787690 10865 log.cc:824] T 7b60b7c168a3455686ff11ed526ff3c2 P 460d8bf451e44c29a2e3066f83241f65: Log is configured to *not* fsync() on all Append() calls
I20260926 16:31:13.788038 10866 tablet_bootstrap.cc:654] T 7b60b7c168a3455686ff11ed526ff3c2 P 9edb1699aa294f3f98bf4ef560ed1e86: Neither blocks nor log segments found. Creating new log.
I20260926 16:31:13.789165 10866 log.cc:824] T 7b60b7c168a3455686ff11ed526ff3c2 P 9edb1699aa294f3f98bf4ef560ed1e86: Log is configured to *not* fsync() on all Append() calls
I20260926 16:31:13.791569 10865 tablet_bootstrap.cc:492] T 7b60b7c168a3455686ff11ed526ff3c2 P 460d8bf451e44c29a2e3066f83241f65: No bootstrap required, opened a new log
I20260926 16:31:13.791884 10866 tablet_bootstrap.cc:492] T 7b60b7c168a3455686ff11ed526ff3c2 P 9edb1699aa294f3f98bf4ef560ed1e86: No bootstrap required, opened a new log
I20260926 16:31:13.791913 10865 ts_tablet_manager.cc:1397] T 7b60b7c168a3455686ff11ed526ff3c2 P 460d8bf451e44c29a2e3066f83241f65: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260926 16:31:13.792138 10866 ts_tablet_manager.cc:1397] T 7b60b7c168a3455686ff11ed526ff3c2 P 9edb1699aa294f3f98bf4ef560ed1e86: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.005s
I20260926 16:31:13.796613 10864 raft_consensus.cc:348] T 7b60b7c168a3455686ff11ed526ff3c2 P 39b6750474b0461f98af5f4b3dd3444b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "460d8bf451e44c29a2e3066f83241f65" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 39703 } } peers { permanent_uuid: "9edb1699aa294f3f98bf4ef560ed1e86" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 38689 } } peers { permanent_uuid: "39b6750474b0461f98af5f4b3dd3444b" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 41389 } }
I20260926 16:31:13.797094 10864 raft_consensus.cc:374] T 7b60b7c168a3455686ff11ed526ff3c2 P 39b6750474b0461f98af5f4b3dd3444b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:31:13.797192 10864 raft_consensus.cc:729] T 7b60b7c168a3455686ff11ed526ff3c2 P 39b6750474b0461f98af5f4b3dd3444b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39b6750474b0461f98af5f4b3dd3444b, State: Initialized, Role: FOLLOWER
I20260926 16:31:13.797919 10864 consensus_queue.cc:260] T 7b60b7c168a3455686ff11ed526ff3c2 P 39b6750474b0461f98af5f4b3dd3444b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "460d8bf451e44c29a2e3066f83241f65" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 39703 } } peers { permanent_uuid: "9edb1699aa294f3f98bf4ef560ed1e86" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 38689 } } peers { permanent_uuid: "39b6750474b0461f98af5f4b3dd3444b" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 41389 } }
I20260926 16:31:13.800210 10866 raft_consensus.cc:348] T 7b60b7c168a3455686ff11ed526ff3c2 P 9edb1699aa294f3f98bf4ef560ed1e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "460d8bf451e44c29a2e3066f83241f65" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 39703 } } peers { permanent_uuid: "9edb1699aa294f3f98bf4ef560ed1e86" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 38689 } } peers { permanent_uuid: "39b6750474b0461f98af5f4b3dd3444b" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 41389 } }
I20260926 16:31:13.800552 10866 raft_consensus.cc:374] T 7b60b7c168a3455686ff11ed526ff3c2 P 9edb1699aa294f3f98bf4ef560ed1e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:31:13.800660 10866 raft_consensus.cc:729] T 7b60b7c168a3455686ff11ed526ff3c2 P 9edb1699aa294f3f98bf4ef560ed1e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9edb1699aa294f3f98bf4ef560ed1e86, State: Initialized, Role: FOLLOWER
I20260926 16:31:13.801273 10866 consensus_queue.cc:260] T 7b60b7c168a3455686ff11ed526ff3c2 P 9edb1699aa294f3f98bf4ef560ed1e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "460d8bf451e44c29a2e3066f83241f65" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 39703 } } peers { permanent_uuid: "9edb1699aa294f3f98bf4ef560ed1e86" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 38689 } } peers { permanent_uuid: "39b6750474b0461f98af5f4b3dd3444b" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 41389 } }
I20260926 16:31:13.802434 10845 heartbeater.cc:503] Master 127.6.33.62:42383 was elected leader, sending a full tablet report...
I20260926 16:31:13.802901 10866 ts_tablet_manager.cc:1428] T 7b60b7c168a3455686ff11ed526ff3c2 P 9edb1699aa294f3f98bf4ef560ed1e86: Time spent starting tablet: real 0.011s user 0.004s sys 0.008s
I20260926 16:31:13.805949 10864 ts_tablet_manager.cc:1428] T 7b60b7c168a3455686ff11ed526ff3c2 P 39b6750474b0461f98af5f4b3dd3444b: Time spent starting tablet: real 0.020s user 0.015s sys 0.004s
I20260926 16:31:13.806035 10865 raft_consensus.cc:348] T 7b60b7c168a3455686ff11ed526ff3c2 P 460d8bf451e44c29a2e3066f83241f65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "460d8bf451e44c29a2e3066f83241f65" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 39703 } } peers { permanent_uuid: "9edb1699aa294f3f98bf4ef560ed1e86" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 38689 } } peers { permanent_uuid: "39b6750474b0461f98af5f4b3dd3444b" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 41389 } }
I20260926 16:31:13.806530 10865 raft_consensus.cc:374] T 7b60b7c168a3455686ff11ed526ff3c2 P 460d8bf451e44c29a2e3066f83241f65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:31:13.806650 10865 raft_consensus.cc:729] T 7b60b7c168a3455686ff11ed526ff3c2 P 460d8bf451e44c29a2e3066f83241f65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 460d8bf451e44c29a2e3066f83241f65, State: Initialized, Role: FOLLOWER
I20260926 16:31:13.807291 10865 consensus_queue.cc:260] T 7b60b7c168a3455686ff11ed526ff3c2 P 460d8bf451e44c29a2e3066f83241f65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "460d8bf451e44c29a2e3066f83241f65" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 39703 } } peers { permanent_uuid: "9edb1699aa294f3f98bf4ef560ed1e86" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 38689 } } peers { permanent_uuid: "39b6750474b0461f98af5f4b3dd3444b" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 41389 } }
I20260926 16:31:13.808990 10713 heartbeater.cc:503] Master 127.6.33.62:42383 was elected leader, sending a full tablet report...
I20260926 16:31:13.809265 10865 ts_tablet_manager.cc:1428] T 7b60b7c168a3455686ff11ed526ff3c2 P 460d8bf451e44c29a2e3066f83241f65: Time spent starting tablet: real 0.017s user 0.015s sys 0.000s
I20260926 16:31:13.887346 10870 raft_consensus.cc:482] T 7b60b7c168a3455686ff11ed526ff3c2 P 39b6750474b0461f98af5f4b3dd3444b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:31:13.887633 10870 raft_consensus.cc:504] T 7b60b7c168a3455686ff11ed526ff3c2 P 39b6750474b0461f98af5f4b3dd3444b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "460d8bf451e44c29a2e3066f83241f65" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 39703 } } peers { permanent_uuid: "9edb1699aa294f3f98bf4ef560ed1e86" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 38689 } } peers { permanent_uuid: "39b6750474b0461f98af5f4b3dd3444b" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 41389 } }
I20260926 16:31:13.889460 10870 leader_election.cc:288] T 7b60b7c168a3455686ff11ed526ff3c2 P 39b6750474b0461f98af5f4b3dd3444b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 460d8bf451e44c29a2e3066f83241f65 (127.6.33.2:39703), 9edb1699aa294f3f98bf4ef560ed1e86 (127.6.33.3:38689)
W20260926 16:31:13.893940 10581 tablet.cc:2369] T 7b60b7c168a3455686ff11ed526ff3c2 P 39b6750474b0461f98af5f4b3dd3444b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:31:13.897491 10873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:13.889706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.2:39703 (local address 127.6.33.1:40039)
0926 16:31:13.890580 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:13.890603 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:13.890788 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:13.891170 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:13.891174 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:13.891201 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:31:13.891664 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:13.891676 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:13.892933 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:13.892939 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:13.895256 (+ 2317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:13.895268 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:13.896712 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:13.896720 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:13.896918 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:13.896965 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 16:31:13.897062 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 16:31:13.897210 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"thread_start_us":420,"threads_started":1}
I20260926 16:31:13.897527 10859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:13.889870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:40039 (local address 127.6.33.2:39703)
0926 16:31:13.890242 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:13.890248 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:13.890745 (+ 497us) server_negotiation.cc:407] Connection header received
0926 16:31:13.890939 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:13.890947 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:13.891030 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:13.891122 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:31:13.891827 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:13.892710 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:13.895410 (+ 2700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:13.896485 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:13.896590 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:13.896662 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:13.897129 (+ 467us) server_negotiation.cc:299] Negotiation successful
0926 16:31:13.897270 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207}
I20260926 16:31:13.897974 10861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:13.890001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:56193 (local address 127.6.33.3:38689)
0926 16:31:13.890413 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:13.890420 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:13.890991 (+ 571us) server_negotiation.cc:407] Connection header received
0926 16:31:13.891197 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:13.891202 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:13.891282 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:13.891392 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:31:13.892232 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:13.893310 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:13.895903 (+ 2593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:13.896944 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:13.897083 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:13.897179 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:13.897591 (+ 412us) server_negotiation.cc:299] Negotiation successful
0926 16:31:13.897728 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268}
I20260926 16:31:13.898059 10874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:13.889816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.3:38689 (local address 127.6.33.1:56193)
0926 16:31:13.890877 (+ 1061us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:13.890895 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:13.891058 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:13.891445 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:13.891449 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:13.891477 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:31:13.892024 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:13.892036 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:13.893509 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:13.893518 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:13.895733 (+ 2215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:13.895746 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:13.897219 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:13.897225 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:13.897409 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:13.897441 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 16:31:13.897549 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 16:31:13.897698 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":931,"thread_start_us":733,"threads_started":1}
W20260926 16:31:13.898155 10714 tablet.cc:2369] T 7b60b7c168a3455686ff11ed526ff3c2 P 460d8bf451e44c29a2e3066f83241f65: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:31:13.898986 10668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b60b7c168a3455686ff11ed526ff3c2" candidate_uuid: "39b6750474b0461f98af5f4b3dd3444b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "460d8bf451e44c29a2e3066f83241f65" is_pre_election: true
I20260926 16:31:13.899087 10800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b60b7c168a3455686ff11ed526ff3c2" candidate_uuid: "39b6750474b0461f98af5f4b3dd3444b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9edb1699aa294f3f98bf4ef560ed1e86" is_pre_election: true
I20260926 16:31:13.899539 10668 raft_consensus.cc:2454] T 7b60b7c168a3455686ff11ed526ff3c2 P 460d8bf451e44c29a2e3066f83241f65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39b6750474b0461f98af5f4b3dd3444b in term 0.
I20260926 16:31:13.899587 10800 raft_consensus.cc:2454] T 7b60b7c168a3455686ff11ed526ff3c2 P 9edb1699aa294f3f98bf4ef560ed1e86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39b6750474b0461f98af5f4b3dd3444b in term 0.
I20260926 16:31:13.900243 10468 leader_election.cc:302] T 7b60b7c168a3455686ff11ed526ff3c2 P 39b6750474b0461f98af5f4b3dd3444b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39b6750474b0461f98af5f4b3dd3444b, 460d8bf451e44c29a2e3066f83241f65; no voters:
I20260926 16:31:13.900782 10870 raft_consensus.cc:2781] T 7b60b7c168a3455686ff11ed526ff3c2 P 39b6750474b0461f98af5f4b3dd3444b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:31:13.900962 10870 raft_consensus.cc:482] T 7b60b7c168a3455686ff11ed526ff3c2 P 39b6750474b0461f98af5f4b3dd3444b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:31:13.901070 10870 raft_consensus.cc:3037] T 7b60b7c168a3455686ff11ed526ff3c2 P 39b6750474b0461f98af5f4b3dd3444b [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:31:13.903475 10870 raft_consensus.cc:504] T 7b60b7c168a3455686ff11ed526ff3c2 P 39b6750474b0461f98af5f4b3dd3444b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "460d8bf451e44c29a2e3066f83241f65" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 39703 } } peers { permanent_uuid: "9edb1699aa294f3f98bf4ef560ed1e86" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 38689 } } peers { permanent_uuid: "39b6750474b0461f98af5f4b3dd3444b" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 41389 } }
I20260926 16:31:13.904099 10870 leader_election.cc:288] T 7b60b7c168a3455686ff11ed526ff3c2 P 39b6750474b0461f98af5f4b3dd3444b [CANDIDATE]: Term 1 election: Requested vote from peers 460d8bf451e44c29a2e3066f83241f65 (127.6.33.2:39703), 9edb1699aa294f3f98bf4ef560ed1e86 (127.6.33.3:38689)
I20260926 16:31:13.904601 10668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b60b7c168a3455686ff11ed526ff3c2" candidate_uuid: "39b6750474b0461f98af5f4b3dd3444b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "460d8bf451e44c29a2e3066f83241f65"
I20260926 16:31:13.904635 10800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b60b7c168a3455686ff11ed526ff3c2" candidate_uuid: "39b6750474b0461f98af5f4b3dd3444b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9edb1699aa294f3f98bf4ef560ed1e86"
I20260926 16:31:13.904843 10668 raft_consensus.cc:3037] T 7b60b7c168a3455686ff11ed526ff3c2 P 460d8bf451e44c29a2e3066f83241f65 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:31:13.904843 10800 raft_consensus.cc:3037] T 7b60b7c168a3455686ff11ed526ff3c2 P 9edb1699aa294f3f98bf4ef560ed1e86 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:31:13.908071 10800 raft_consensus.cc:2454] T 7b60b7c168a3455686ff11ed526ff3c2 P 9edb1699aa294f3f98bf4ef560ed1e86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39b6750474b0461f98af5f4b3dd3444b in term 1.
I20260926 16:31:13.908097 10668 raft_consensus.cc:2454] T 7b60b7c168a3455686ff11ed526ff3c2 P 460d8bf451e44c29a2e3066f83241f65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39b6750474b0461f98af5f4b3dd3444b in term 1.
I20260926 16:31:13.908560 10470 leader_election.cc:302] T 7b60b7c168a3455686ff11ed526ff3c2 P 39b6750474b0461f98af5f4b3dd3444b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 39b6750474b0461f98af5f4b3dd3444b, 460d8bf451e44c29a2e3066f83241f65, 9edb1699aa294f3f98bf4ef560ed1e86; no voters:
I20260926 16:31:13.908880 10870 raft_consensus.cc:2781] T 7b60b7c168a3455686ff11ed526ff3c2 P 39b6750474b0461f98af5f4b3dd3444b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:31:13.909343 10870 raft_consensus.cc:686] T 7b60b7c168a3455686ff11ed526ff3c2 P 39b6750474b0461f98af5f4b3dd3444b [term 1 LEADER]: Becoming Leader. State: Replica: 39b6750474b0461f98af5f4b3dd3444b, State: Running, Role: LEADER
I20260926 16:31:13.909816 10870 consensus_queue.cc:237] T 7b60b7c168a3455686ff11ed526ff3c2 P 39b6750474b0461f98af5f4b3dd3444b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "460d8bf451e44c29a2e3066f83241f65" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 39703 } } peers { permanent_uuid: "9edb1699aa294f3f98bf4ef560ed1e86" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 38689 } } peers { permanent_uuid: "39b6750474b0461f98af5f4b3dd3444b" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 41389 } }
I20260926 16:31:13.914162 10391 catalog_manager.cc:5495] T 7b60b7c168a3455686ff11ed526ff3c2 P 39b6750474b0461f98af5f4b3dd3444b reported cstate change: term changed from 0 to 1, leader changed from <none> to 39b6750474b0461f98af5f4b3dd3444b (127.6.33.1). New cstate: current_term: 1 leader_uuid: "39b6750474b0461f98af5f4b3dd3444b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "460d8bf451e44c29a2e3066f83241f65" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 39703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9edb1699aa294f3f98bf4ef560ed1e86" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 38689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39b6750474b0461f98af5f4b3dd3444b" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 41389 } health_report { overall_health: HEALTHY } } }
W20260926 16:31:13.919414 10846 tablet.cc:2369] T 7b60b7c168a3455686ff11ed526ff3c2 P 9edb1699aa294f3f98bf4ef560ed1e86: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:31:13.951609 10863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:13.946298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51654 (local address 127.6.33.1:41389)
0926 16:31:13.946549 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:13.946557 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:13.946579 (+ 22us) server_negotiation.cc:407] Connection header received
0926 16:31:13.946662 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:13.946666 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:13.946737 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:13.946894 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:31:13.947289 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:13.948042 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:13.949437 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:13.949859 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:13.949993 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:13.950493 (+ 500us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:31:13.951018 (+ 525us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:31:13.951181 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:13.951260 (+ 79us) server_negotiation.cc:299] Negotiation successful
0926 16:31:13.951341 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260926 16:31:13.960032 10668 raft_consensus.cc:1261] T 7b60b7c168a3455686ff11ed526ff3c2 P 460d8bf451e44c29a2e3066f83241f65 [term 1 FOLLOWER]: Refusing update from remote peer 39b6750474b0461f98af5f4b3dd3444b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:31:13.960032 10800 raft_consensus.cc:1261] T 7b60b7c168a3455686ff11ed526ff3c2 P 9edb1699aa294f3f98bf4ef560ed1e86 [term 1 FOLLOWER]: Refusing update from remote peer 39b6750474b0461f98af5f4b3dd3444b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:31:13.961138 10875 consensus_queue.cc:1035] T 7b60b7c168a3455686ff11ed526ff3c2 P 39b6750474b0461f98af5f4b3dd3444b [LEADER]: Connected to new peer: Peer: permanent_uuid: "460d8bf451e44c29a2e3066f83241f65" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 39703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:31:13.961454 10870 consensus_queue.cc:1035] T 7b60b7c168a3455686ff11ed526ff3c2 P 39b6750474b0461f98af5f4b3dd3444b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9edb1699aa294f3f98bf4ef560ed1e86" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 38689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:31:13.972308 10878 mvcc.cc:205] Tried to move back new op lower bound from 7333643362129637376 to 7333643361938247680. Current Snapshot: MvccSnapshot[applied={T|T < 7333643362129637376}]
I20260926 16:31:13.974599 10879 mvcc.cc:205] Tried to move back new op lower bound from 7333643362129637376 to 7333643361938247680. Current Snapshot: MvccSnapshot[applied={T|T < 7333643362129637376}]
I20260926 16:31:13.986759 10391 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:31:13.987035 10391 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:31:13.990584 10391 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1f17986b5f1a4c079f1b74a9a7953b85: Sending DeleteTablet for 3 replicas of tablet 7b60b7c168a3455686ff11ed526ff3c2
I20260926 16:31:13.991804 10648 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b60b7c168a3455686ff11ed526ff3c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:31:13 UTC) from {username='kudu', principal='kudu/127.6.33.62@KRBTEST.COM'} at 127.0.0.1:58776
I20260926 16:31:13.992015 10515 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b60b7c168a3455686ff11ed526ff3c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:31:13 UTC) from {username='kudu', principal='kudu/127.6.33.62@KRBTEST.COM'} at 127.0.0.1:51644
I20260926 16:31:13.992146 10780 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b60b7c168a3455686ff11ed526ff3c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:31:13 UTC) from {username='kudu', principal='kudu/127.6.33.62@KRBTEST.COM'} at 127.0.0.1:39784
I20260926 16:31:13.994206 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 10453
I20260926 16:31:14.004433 10887 tablet_replica.cc:330] T 7b60b7c168a3455686ff11ed526ff3c2 P 460d8bf451e44c29a2e3066f83241f65: stopping tablet replica
I20260926 16:31:14.005198 10887 raft_consensus.cc:2229] T 7b60b7c168a3455686ff11ed526ff3c2 P 460d8bf451e44c29a2e3066f83241f65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:31:14.005995 10887 raft_consensus.cc:2258] T 7b60b7c168a3455686ff11ed526ff3c2 P 460d8bf451e44c29a2e3066f83241f65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:31:14.008951 10887 ts_tablet_manager.cc:1905] T 7b60b7c168a3455686ff11ed526ff3c2 P 460d8bf451e44c29a2e3066f83241f65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:31:14.018455 10887 ts_tablet_manager.cc:1918] T 7b60b7c168a3455686ff11ed526ff3c2 P 460d8bf451e44c29a2e3066f83241f65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:31:14.018810 10887 log.cc:1196] T 7b60b7c168a3455686ff11ed526ff3c2 P 460d8bf451e44c29a2e3066f83241f65: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/ts-1/wal/wals/7b60b7c168a3455686ff11ed526ff3c2
I20260926 16:31:14.019556 10887 ts_tablet_manager.cc:1939] T 7b60b7c168a3455686ff11ed526ff3c2 P 460d8bf451e44c29a2e3066f83241f65: Deleting consensus metadata
I20260926 16:31:14.021768 10376 catalog_manager.cc:4841] TS 460d8bf451e44c29a2e3066f83241f65 (127.6.33.2:39703): tablet 7b60b7c168a3455686ff11ed526ff3c2 (table test-table [id=8fd88584a04d4eb58e3212323c7daa3e]) successfully deleted
I20260926 16:31:14.024019 10888 tablet_replica.cc:330] T 7b60b7c168a3455686ff11ed526ff3c2 P 39b6750474b0461f98af5f4b3dd3444b: stopping tablet replica
I20260926 16:31:14.024559 10888 raft_consensus.cc:2229] T 7b60b7c168a3455686ff11ed526ff3c2 P 39b6750474b0461f98af5f4b3dd3444b [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:31:14.025331 10888 raft_consensus.cc:2258] T 7b60b7c168a3455686ff11ed526ff3c2 P 39b6750474b0461f98af5f4b3dd3444b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:31:14.027616 10888 ts_tablet_manager.cc:1905] T 7b60b7c168a3455686ff11ed526ff3c2 P 39b6750474b0461f98af5f4b3dd3444b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:31:14.028839 10889 tablet_replica.cc:330] T 7b60b7c168a3455686ff11ed526ff3c2 P 9edb1699aa294f3f98bf4ef560ed1e86: stopping tablet replica
I20260926 16:31:14.029352 10889 raft_consensus.cc:2229] T 7b60b7c168a3455686ff11ed526ff3c2 P 9edb1699aa294f3f98bf4ef560ed1e86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:31:14.029912 10889 raft_consensus.cc:2258] T 7b60b7c168a3455686ff11ed526ff3c2 P 9edb1699aa294f3f98bf4ef560ed1e86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:31:14.031760 10889 ts_tablet_manager.cc:1905] T 7b60b7c168a3455686ff11ed526ff3c2 P 9edb1699aa294f3f98bf4ef560ed1e86: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 16:31:14 dist-test-slave-khd3 krb5kdc[10348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.1@KRBTEST.COM
I20260926 16:31:14.036522 10888 ts_tablet_manager.cc:1918] T 7b60b7c168a3455686ff11ed526ff3c2 P 39b6750474b0461f98af5f4b3dd3444b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:31:14.036705 10888 log.cc:1196] T 7b60b7c168a3455686ff11ed526ff3c2 P 39b6750474b0461f98af5f4b3dd3444b: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/ts-0/wal/wals/7b60b7c168a3455686ff11ed526ff3c2
I20260926 16:31:14.037276 10888 ts_tablet_manager.cc:1939] T 7b60b7c168a3455686ff11ed526ff3c2 P 39b6750474b0461f98af5f4b3dd3444b: Deleting consensus metadata
I20260926 16:31:14.039194 10889 ts_tablet_manager.cc:1918] T 7b60b7c168a3455686ff11ed526ff3c2 P 9edb1699aa294f3f98bf4ef560ed1e86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:31:14.039204 10378 catalog_manager.cc:4841] TS 39b6750474b0461f98af5f4b3dd3444b (127.6.33.1:41389): tablet 7b60b7c168a3455686ff11ed526ff3c2 (table test-table [id=8fd88584a04d4eb58e3212323c7daa3e]) successfully deleted
I20260926 16:31:14.039456 10889 log.cc:1196] T 7b60b7c168a3455686ff11ed526ff3c2 P 9edb1699aa294f3f98bf4ef560ed1e86: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440233850748-6276-0/minicluster-data/ts-2/wal/wals/7b60b7c168a3455686ff11ed526ff3c2
I20260926 16:31:14.039944 10889 ts_tablet_manager.cc:1939] T 7b60b7c168a3455686ff11ed526ff3c2 P 9edb1699aa294f3f98bf4ef560ed1e86: Deleting consensus metadata
I20260926 16:31:14.041534 10863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:13.997400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51670 (local address 127.6.33.1:41389)
0926 16:31:13.997649 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:13.997658 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:14.020338 (+ 22680us) server_negotiation.cc:407] Connection header received
0926 16:31:14.020910 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:14.020920 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:14.021007 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:14.021095 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:14.022774 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:14.024241 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:14.026712 (+ 2471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:14.027374 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:14.027470 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:14.033308 (+ 5838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:14.033335 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:14.033354 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:14.033412 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:14.037626 (+ 4214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:14.038568 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:14.038578 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:14.038581 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:14.038800 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:14.039292 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:14.039298 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:14.039304 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:14.039774 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:14.040045 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:14.040645 (+ 600us) server_negotiation.cc:299] Negotiation successful
0926 16:31:14.040953 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 16:31:14.041795 10378 catalog_manager.cc:4841] TS 9edb1699aa294f3f98bf4ef560ed1e86 (127.6.33.3:38689): tablet 7b60b7c168a3455686ff11ed526ff3c2 (table test-table [id=8fd88584a04d4eb58e3212323c7daa3e]) successfully deleted
I20260926 16:31:14.042261 10575 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:31:14.149816 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 10453
I20260926 16:31:14.169358 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 10586
Sep 26 16:31:14 dist-test-slave-khd3 krb5kdc[10348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.2@KRBTEST.COM
I20260926 16:31:14.184530 10859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:14.170261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58792 (local address 127.6.33.2:39703)
0926 16:31:14.170419 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:14.170424 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:14.170441 (+ 17us) server_negotiation.cc:407] Connection header received
0926 16:31:14.170493 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:14.170497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:14.170580 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:14.170670 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:14.172097 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:14.172971 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:14.174305 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:14.174722 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:14.174822 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:14.179085 (+ 4263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:14.179099 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:14.179115 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:14.179147 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:14.182109 (+ 2962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:14.182790 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:14.182798 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:14.182800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:14.182932 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:14.183257 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:14.183260 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:14.183262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:14.183620 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:14.183823 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:14.184097 (+ 274us) server_negotiation.cc:299] Negotiation successful
0926 16:31:14.184259 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 16:31:14.185240 10708 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:31:14.275385 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 10586
I20260926 16:31:14.292258 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 10718
Sep 26 16:31:14 dist-test-slave-khd3 krb5kdc[10348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.3@KRBTEST.COM
I20260926 16:31:14.307912 10861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:14.293051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39800 (local address 127.6.33.3:38689)
0926 16:31:14.293252 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:14.293258 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:14.293353 (+ 95us) server_negotiation.cc:407] Connection header received
0926 16:31:14.293559 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:14.293564 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:14.293646 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:14.293765 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:14.295047 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:14.295852 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:14.297262 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:14.297656 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:14.297785 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:14.301908 (+ 4123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:14.301923 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:14.301939 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:14.301975 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:14.305139 (+ 3164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:14.305809 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:14.305824 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:14.305827 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:14.305945 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:14.306311 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:14.306317 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:14.306320 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:14.306686 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:14.306989 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:14.307363 (+ 374us) server_negotiation.cc:299] Negotiation successful
0926 16:31:14.307593 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 16:31:14.308575 10840 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:31:14.396731 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 10718
I20260926 16:31:14.413182 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 10361
I20260926 16:31:14.414369 10422 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:31:14.521055 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 10361
2026-09-26T16:31:14Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4190 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-26T16:31:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:31:14Z Disabled control of system clock
I20260926 16:31:14.590008 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:43931
--webserver_interface=127.6.33.62
--webserver_port=0
--builtin_ntp_servers=127.6.33.20:44681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.33.62:43931
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 16:31:14.924973 10898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:31:14.925283 10898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:31:14.925444 10898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:31:14.933378 10898 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:31:14.933486 10898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:31:14.933557 10898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:31:14.933642 10898 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:31:14.933723 10898 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:31:14.943893 10898 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:44681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.33.62:43931
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:43931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:31:14.945348 10898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:31:14.947134 10898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:31:14.959110 10904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:14.959219 10906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:14.959173 10903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:31:14.960018 10898 server_base.cc:1034] running on GCE node
I20260926 16:31:14.960927 10898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:31:14.962322 10898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:31:14.963574 10898 hybrid_clock.cc:648] HybridClock initialized: now 1790440274963508 us; error 55 us; skew 500 ppm
I20260926 16:31:14.963955 10898 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:31:14.966360 10898 webserver.cc:460] Webserver started at http://127.6.33.62:43803/ using document root <none> and password file <none>
I20260926 16:31:14.967087 10898 fs_manager.cc:359] Metadata directory not provided
I20260926 16:31:14.967221 10898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:31:14.967545 10898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:31:14.971463 10898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/master-0/data/instance:
uuid: "b7bc229df4be40f9b01cfab406d2ff69"
format_stamp: "Formatted at 2026-09-26 16:31:14 on dist-test-slave-khd3"
server_key: "d22ca4a5928574aa3b759373eef7d8c1"
server_key_iv: "d421fb346a23df5f35572a562671dae3"
server_key_version: "encryptionkey@0"
I20260926 16:31:14.972523 10898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/master-0/wal/instance:
uuid: "b7bc229df4be40f9b01cfab406d2ff69"
format_stamp: "Formatted at 2026-09-26 16:31:14 on dist-test-slave-khd3"
server_key: "d22ca4a5928574aa3b759373eef7d8c1"
server_key_iv: "d421fb346a23df5f35572a562671dae3"
server_key_version: "encryptionkey@0"
I20260926 16:31:14.978356 10898 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 16:31:14.982056 10912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:14.983541 10898 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:31:14.983731 10898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/master-0/wal
uuid: "b7bc229df4be40f9b01cfab406d2ff69"
format_stamp: "Formatted at 2026-09-26 16:31:14 on dist-test-slave-khd3"
server_key: "d22ca4a5928574aa3b759373eef7d8c1"
server_key_iv: "d421fb346a23df5f35572a562671dae3"
server_key_version: "encryptionkey@0"
I20260926 16:31:14.983922 10898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:31:14.989696 10898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:31:14.990420 10898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:31:14.990674 10898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:31:15.008924 10898 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.62:43931
I20260926 16:31:15.009011 10963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.62:43931 every 8 connection(s)
I20260926 16:31:15.010335 10898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
I20260926 16:31:15.014243 10964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:31:15.019873 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 10898
I20260926 16:31:15.020104 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/master-0/wal/instance
I20260926 16:31:15.020964 6276 external_mini_cluster.cc:1428] Setting key f8068e8fb8af5e80115fb959c4ddf2eb
I20260926 16:31:15.026437 10964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7bc229df4be40f9b01cfab406d2ff69: Bootstrap starting.
I20260926 16:31:15.029997 10964 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7bc229df4be40f9b01cfab406d2ff69: Neither blocks nor log segments found. Creating new log.
I20260926 16:31:15.030690 10967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:15.023472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55424 (local address 127.6.33.62:43931)
0926 16:31:15.024661 (+ 1189us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:15.024673 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:15.024718 (+ 45us) server_negotiation.cc:407] Connection header received
0926 16:31:15.026476 (+ 1758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:15.026501 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:15.027045 (+ 544us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:15.027455 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:15.028685 (+ 1230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:15.028704 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:15.028719 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:31:15.028743 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:15.028892 (+ 149us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:31:15.028917 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:15.029178 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:15.029330 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 16:31:15.029605 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":675,"thread_start_us":365,"threads_started":1}
I20260926 16:31:15.031180 10964 log.cc:824] T 00000000000000000000000000000000 P b7bc229df4be40f9b01cfab406d2ff69: Log is configured to *not* fsync() on all Append() calls
I20260926 16:31:15.034821 10964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7bc229df4be40f9b01cfab406d2ff69: No bootstrap required, opened a new log
I20260926 16:31:15.042555 10964 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7bc229df4be40f9b01cfab406d2ff69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7bc229df4be40f9b01cfab406d2ff69" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 43931 } }
I20260926 16:31:15.042946 10964 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7bc229df4be40f9b01cfab406d2ff69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:31:15.043016 10964 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7bc229df4be40f9b01cfab406d2ff69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7bc229df4be40f9b01cfab406d2ff69, State: Initialized, Role: FOLLOWER
I20260926 16:31:15.043658 10964 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7bc229df4be40f9b01cfab406d2ff69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7bc229df4be40f9b01cfab406d2ff69" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 43931 } }
I20260926 16:31:15.043888 10964 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7bc229df4be40f9b01cfab406d2ff69 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:31:15.043984 10964 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7bc229df4be40f9b01cfab406d2ff69 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:31:15.044121 10964 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7bc229df4be40f9b01cfab406d2ff69 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:31:15.046741 10964 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7bc229df4be40f9b01cfab406d2ff69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7bc229df4be40f9b01cfab406d2ff69" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 43931 } }
I20260926 16:31:15.047267 10964 leader_election.cc:302] T 00000000000000000000000000000000 P b7bc229df4be40f9b01cfab406d2ff69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7bc229df4be40f9b01cfab406d2ff69; no voters:
I20260926 16:31:15.047892 10964 leader_election.cc:288] T 00000000000000000000000000000000 P b7bc229df4be40f9b01cfab406d2ff69 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:31:15.048095 10969 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7bc229df4be40f9b01cfab406d2ff69 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:31:15.048908 10969 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7bc229df4be40f9b01cfab406d2ff69 [term 1 LEADER]: Becoming Leader. State: Replica: b7bc229df4be40f9b01cfab406d2ff69, State: Running, Role: LEADER
I20260926 16:31:15.049480 10969 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7bc229df4be40f9b01cfab406d2ff69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7bc229df4be40f9b01cfab406d2ff69" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 43931 } }
I20260926 16:31:15.050140 10964 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7bc229df4be40f9b01cfab406d2ff69 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:31:15.052636 10971 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7bc229df4be40f9b01cfab406d2ff69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7bc229df4be40f9b01cfab406d2ff69. Latest consensus state: current_term: 1 leader_uuid: "b7bc229df4be40f9b01cfab406d2ff69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7bc229df4be40f9b01cfab406d2ff69" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 43931 } } }
I20260926 16:31:15.052474 10970 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7bc229df4be40f9b01cfab406d2ff69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b7bc229df4be40f9b01cfab406d2ff69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7bc229df4be40f9b01cfab406d2ff69" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 43931 } } }
I20260926 16:31:15.053036 10971 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7bc229df4be40f9b01cfab406d2ff69 [sys.catalog]: This master's current role is: LEADER
I20260926 16:31:15.053040 10970 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7bc229df4be40f9b01cfab406d2ff69 [sys.catalog]: This master's current role is: LEADER
I20260926 16:31:15.054253 10975 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:31:15.061987 10975 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:31:15.071727 10975 catalog_manager.cc:1347] Generated new cluster ID: d39070ed0c6043f9b44e9439807fd97a
I20260926 16:31:15.071874 10975 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:31:15.082950 10975 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:31:15.083591 10975 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:31:15.095410 10975 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b7bc229df4be40f9b01cfab406d2ff69: Generated new TSK 0
I20260926 16:31:15.096361 10975 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:31:15.104526 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.1:0
--local_ip_for_outbound_sockets=127.6.33.1
--webserver_interface=127.6.33.1
--webserver_port=0
--tserver_master_addrs=127.6.33.62:43931
--builtin_ntp_servers=127.6.33.20:44681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 16:31:15.438264 10988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:31:15.438591 10988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:31:15.438735 10988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:31:15.446781 10988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:31:15.446897 10988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:31:15.447088 10988 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.1
I20260926 16:31:15.457372 10988 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:44681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.33.1
--webserver_port=0
--tserver_master_addrs=127.6.33.62:43931
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.1
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:31:15.458729 10988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:31:15.460557 10988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:31:15.474642 10994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:15.474670 10996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:15.474709 10993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:31:15.475500 10988 server_base.cc:1034] running on GCE node
I20260926 16:31:15.476143 10988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:31:15.477224 10988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:31:15.478459 10988 hybrid_clock.cc:648] HybridClock initialized: now 1790440275478396 us; error 48 us; skew 500 ppm
I20260926 16:31:15.478850 10988 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:31:15.481304 10988 webserver.cc:460] Webserver started at http://127.6.33.1:45833/ using document root <none> and password file <none>
I20260926 16:31:15.482060 10988 fs_manager.cc:359] Metadata directory not provided
I20260926 16:31:15.482189 10988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:31:15.482507 10988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:31:15.485441 10988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/ts-0/data/instance:
uuid: "d46b9aafbf004df0b9810c307a502277"
format_stamp: "Formatted at 2026-09-26 16:31:15 on dist-test-slave-khd3"
server_key: "ad28e7c06413ca34c39ff180c9fa1bab"
server_key_iv: "88d4ea3ffb410b2bdef224bbded954ca"
server_key_version: "encryptionkey@0"
I20260926 16:31:15.486207 10988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/ts-0/wal/instance:
uuid: "d46b9aafbf004df0b9810c307a502277"
format_stamp: "Formatted at 2026-09-26 16:31:15 on dist-test-slave-khd3"
server_key: "ad28e7c06413ca34c39ff180c9fa1bab"
server_key_iv: "88d4ea3ffb410b2bdef224bbded954ca"
server_key_version: "encryptionkey@0"
I20260926 16:31:15.491515 10988 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 16:31:15.495146 11002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:15.496562 10988 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 16:31:15.496742 10988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/ts-0/wal
uuid: "d46b9aafbf004df0b9810c307a502277"
format_stamp: "Formatted at 2026-09-26 16:31:15 on dist-test-slave-khd3"
server_key: "ad28e7c06413ca34c39ff180c9fa1bab"
server_key_iv: "88d4ea3ffb410b2bdef224bbded954ca"
server_key_version: "encryptionkey@0"
I20260926 16:31:15.497030 10988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:31:15.502620 10988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:31:15.503379 10988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:31:15.503647 10988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:31:15.504542 10988 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:31:15.505944 10988 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:31:15.506057 10988 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:15.506182 10988 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:31:15.506264 10988 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:15.539547 10988 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.1:37681
I20260926 16:31:15.539633 11114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.1:37681 every 8 connection(s)
I20260926 16:31:15.540947 10988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
I20260926 16:31:15.544059 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 10988
I20260926 16:31:15.544282 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/ts-0/wal/instance
I20260926 16:31:15.545140 6276 external_mini_cluster.cc:1428] Setting key 8702cdea4e39e01ee9b5dbaae3d03181
I20260926 16:31:15.547950 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.2:0
--local_ip_for_outbound_sockets=127.6.33.2
--webserver_interface=127.6.33.2
--webserver_port=0
--tserver_master_addrs=127.6.33.62:43931
--builtin_ntp_servers=127.6.33.20:44681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 16:31:15.550004 11117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:15.543532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:40303 (local address 127.6.33.62:43931)
0926 16:31:15.543995 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:15.544000 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:15.545518 (+ 1518us) server_negotiation.cc:407] Connection header received
0926 16:31:15.547430 (+ 1912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:15.547435 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:15.547507 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:15.547625 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:15.549197 (+ 1572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:15.549203 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:15.549208 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:31:15.549233 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:15.549277 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:31:15.549289 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:15.549420 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:15.549657 (+ 237us) server_negotiation.cc:299] Negotiation successful
0926 16:31:15.549734 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":252,"threads_started":1}
I20260926 16:31:15.551168 11118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:15.544207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:43931 (local address 127.6.33.1:40303)
0926 16:31:15.545346 (+ 1139us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:15.545386 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:15.547163 (+ 1777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:15.547878 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:15.547887 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:15.547996 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:15.548880 (+ 884us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:31:15.548883 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:15.548926 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:31:15.549022 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:31:15.549391 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:31:15.549403 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:31:15.549412 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 16:31:15.549780 (+ 368us) client_negotiation.cc:240] Negotiation successful
0926 16:31:15.550070 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":336,"threads_started":1}
I20260926 16:31:15.552914 11115 heartbeater.cc:347] Connected to a master server at 127.6.33.62:43931
I20260926 16:31:15.553294 11115 heartbeater.cc:464] Registering TS with master...
I20260926 16:31:15.554090 11115 heartbeater.cc:511] Master 127.6.33.62:43931 requested a full tablet report, sending...
I20260926 16:31:15.556600 10929 ts_manager.cc:194] Registered new tserver with Master: d46b9aafbf004df0b9810c307a502277 (127.6.33.1:37681)
W20260926 16:31:15.883658 11119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:31:15.883944 11119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:31:15.884070 11119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:31:15.891425 11119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:31:15.891526 11119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:31:15.891633 11119 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.2
I20260926 16:31:15.900594 11119 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:44681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.33.2
--webserver_port=0
--tserver_master_addrs=127.6.33.62:43931
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.2
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:31:15.901893 11119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:31:15.903580 11119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:31:15.916334 11125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:15.916380 11124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:15.916501 11127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:31:15.917383 11119 server_base.cc:1034] running on GCE node
I20260926 16:31:15.917980 11119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:31:15.918982 11119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:31:15.920233 11119 hybrid_clock.cc:648] HybridClock initialized: now 1790440275920165 us; error 60 us; skew 500 ppm
I20260926 16:31:15.920595 11119 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:31:15.923026 11119 webserver.cc:460] Webserver started at http://127.6.33.2:33795/ using document root <none> and password file <none>
I20260926 16:31:15.923729 11119 fs_manager.cc:359] Metadata directory not provided
I20260926 16:31:15.923857 11119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:31:15.924178 11119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:31:15.927080 11119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/ts-1/data/instance:
uuid: "f7d7d4f67f85465ca3999dbda1830b67"
format_stamp: "Formatted at 2026-09-26 16:31:15 on dist-test-slave-khd3"
server_key: "58d16da2861dc3210e676dfb361f9b65"
server_key_iv: "e8568c0e48f64ee7bf789dc32a6bc1a0"
server_key_version: "encryptionkey@0"
I20260926 16:31:15.927834 11119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/ts-1/wal/instance:
uuid: "f7d7d4f67f85465ca3999dbda1830b67"
format_stamp: "Formatted at 2026-09-26 16:31:15 on dist-test-slave-khd3"
server_key: "58d16da2861dc3210e676dfb361f9b65"
server_key_iv: "e8568c0e48f64ee7bf789dc32a6bc1a0"
server_key_version: "encryptionkey@0"
I20260926 16:31:15.932984 11119 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 16:31:15.936393 11133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:15.937767 11119 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:31:15.937932 11119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/ts-1/wal
uuid: "f7d7d4f67f85465ca3999dbda1830b67"
format_stamp: "Formatted at 2026-09-26 16:31:15 on dist-test-slave-khd3"
server_key: "58d16da2861dc3210e676dfb361f9b65"
server_key_iv: "e8568c0e48f64ee7bf789dc32a6bc1a0"
server_key_version: "encryptionkey@0"
I20260926 16:31:15.938170 11119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:31:15.943749 11119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:31:15.944543 11119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:31:15.944870 11119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:31:15.945757 11119 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:31:15.947108 11119 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:31:15.947224 11119 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:15.947337 11119 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:31:15.947419 11119 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:15.980964 11119 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.2:36925
I20260926 16:31:15.981019 11245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.2:36925 every 8 connection(s)
I20260926 16:31:15.982297 11119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
I20260926 16:31:15.989909 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 11119
I20260926 16:31:15.990118 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/ts-1/wal/instance
I20260926 16:31:15.990687 6276 external_mini_cluster.cc:1428] Setting key 72fb4788ac37e90b244d47d11c35b14f
I20260926 16:31:15.990702 11117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:15.984966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:58357 (local address 127.6.33.62:43931)
0926 16:31:15.985176 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:15.985182 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:15.986697 (+ 1515us) server_negotiation.cc:407] Connection header received
0926 16:31:15.988383 (+ 1686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:15.988388 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:15.988465 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:15.988602 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:15.989901 (+ 1299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:15.989908 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:15.989912 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:31:15.989943 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:15.989984 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:31:15.990002 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:15.990142 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:15.990355 (+ 213us) server_negotiation.cc:299] Negotiation successful
0926 16:31:15.990415 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 16:31:15.991657 11248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:15.985599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:43931 (local address 127.6.33.2:58357)
0926 16:31:15.986556 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:15.986592 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:15.988138 (+ 1546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:15.988796 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:15.988812 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:15.988906 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:15.989611 (+ 705us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:31:15.989614 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:15.989651 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:31:15.989734 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:31:15.990114 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:31:15.990123 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:31:15.990132 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 16:31:15.990392 (+ 260us) client_negotiation.cc:240] Negotiation successful
0926 16:31:15.990598 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":306,"threads_started":1}
I20260926 16:31:15.993134 11246 heartbeater.cc:347] Connected to a master server at 127.6.33.62:43931
I20260926 16:31:15.993454 11246 heartbeater.cc:464] Registering TS with master...
I20260926 16:31:15.993760 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.3:0
--local_ip_for_outbound_sockets=127.6.33.3
--webserver_interface=127.6.33.3
--webserver_port=0
--tserver_master_addrs=127.6.33.62:43931
--builtin_ntp_servers=127.6.33.20:44681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 16:31:15.994203 11246 heartbeater.cc:511] Master 127.6.33.62:43931 requested a full tablet report, sending...
I20260926 16:31:15.995836 10929 ts_manager.cc:194] Registered new tserver with Master: f7d7d4f67f85465ca3999dbda1830b67 (127.6.33.2:36925)
W20260926 16:31:16.328024 11249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:31:16.328325 11249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:31:16.328487 11249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:31:16.335965 11249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:31:16.336076 11249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:31:16.336249 11249 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.3
I20260926 16:31:16.345908 11249 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:44681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.33.3
--webserver_port=0
--tserver_master_addrs=127.6.33.62:43931
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.3
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:31:16.347227 11249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:31:16.349069 11249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:31:16.362360 11255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:16.362493 11254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:16.362318 11257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:31:16.363132 11249 server_base.cc:1034] running on GCE node
I20260926 16:31:16.363826 11249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:31:16.364832 11249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:31:16.366062 11249 hybrid_clock.cc:648] HybridClock initialized: now 1790440276365995 us; error 70 us; skew 500 ppm
I20260926 16:31:16.366379 11249 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:31:16.368706 11249 webserver.cc:460] Webserver started at http://127.6.33.3:34161/ using document root <none> and password file <none>
I20260926 16:31:16.369462 11249 fs_manager.cc:359] Metadata directory not provided
I20260926 16:31:16.369593 11249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:31:16.369901 11249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:31:16.372735 11249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/ts-2/data/instance:
uuid: "410ac670fa3846e5bb405c0f009253e4"
format_stamp: "Formatted at 2026-09-26 16:31:16 on dist-test-slave-khd3"
server_key: "85633bd4d57734e02021ace2a7ead75d"
server_key_iv: "8cdd368df1ed3cb11dc3907691c90e18"
server_key_version: "encryptionkey@0"
I20260926 16:31:16.373589 11249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/ts-2/wal/instance:
uuid: "410ac670fa3846e5bb405c0f009253e4"
format_stamp: "Formatted at 2026-09-26 16:31:16 on dist-test-slave-khd3"
server_key: "85633bd4d57734e02021ace2a7ead75d"
server_key_iv: "8cdd368df1ed3cb11dc3907691c90e18"
server_key_version: "encryptionkey@0"
I20260926 16:31:16.378706 11249 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260926 16:31:16.382094 11263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:16.383481 11249 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:31:16.383646 11249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/ts-2/wal
uuid: "410ac670fa3846e5bb405c0f009253e4"
format_stamp: "Formatted at 2026-09-26 16:31:16 on dist-test-slave-khd3"
server_key: "85633bd4d57734e02021ace2a7ead75d"
server_key_iv: "8cdd368df1ed3cb11dc3907691c90e18"
server_key_version: "encryptionkey@0"
I20260926 16:31:16.383864 11249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:31:16.389617 11249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:31:16.390390 11249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:31:16.390659 11249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:31:16.391561 11249 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:31:16.393060 11249 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:31:16.393189 11249 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:16.393306 11249 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:31:16.393396 11249 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:16.427604 11249 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.3:33443
I20260926 16:31:16.427690 11375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.3:33443 every 8 connection(s)
I20260926 16:31:16.428992 11249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
I20260926 16:31:16.434051 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 11249
I20260926 16:31:16.434245 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/ts-2/wal/instance
I20260926 16:31:16.434643 6276 external_mini_cluster.cc:1428] Setting key af4911feff5d1eca0a0b86c88dc0fd77
I20260926 16:31:16.437824 11117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:16.431449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.3:33095 (local address 127.6.33.62:43931)
0926 16:31:16.431643 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:16.431648 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:16.433146 (+ 1498us) server_negotiation.cc:407] Connection header received
0926 16:31:16.434934 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:16.434939 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:16.435011 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:16.435118 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:16.436447 (+ 1329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:16.436453 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:16.436458 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:31:16.436476 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:16.436514 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:31:16.436532 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:16.436651 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:16.437461 (+ 810us) server_negotiation.cc:299] Negotiation successful
0926 16:31:16.437536 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 16:31:16.438763 11378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:16.432043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:43931 (local address 127.6.33.3:33095)
0926 16:31:16.432980 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:16.433038 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:16.434651 (+ 1613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:16.435338 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:16.435346 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:16.435437 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:16.436171 (+ 734us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:31:16.436174 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:16.436209 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:31:16.436290 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:31:16.437271 (+ 981us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:31:16.437278 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:31:16.437285 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 16:31:16.437544 (+ 259us) client_negotiation.cc:240] Negotiation successful
0926 16:31:16.437846 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":259,"threads_started":1}
I20260926 16:31:16.440081 11376 heartbeater.cc:347] Connected to a master server at 127.6.33.62:43931
I20260926 16:31:16.440326 11376 heartbeater.cc:464] Registering TS with master...
I20260926 16:31:16.440932 11376 heartbeater.cc:511] Master 127.6.33.62:43931 requested a full tablet report, sending...
I20260926 16:31:16.442305 10929 ts_manager.cc:194] Registered new tserver with Master: 410ac670fa3846e5bb405c0f009253e4 (127.6.33.3:33443)
I20260926 16:31:16.450150 6276 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:31:16.457336 11117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:16.454637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57034 (local address 127.6.33.62:43931)
0926 16:31:16.454852 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:16.454862 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:16.455343 (+ 481us) server_negotiation.cc:407] Connection header received
0926 16:31:16.455497 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:16.455500 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:16.455582 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:16.455656 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:16.456714 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:16.456718 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:16.456722 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:31:16.456737 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:16.456815 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:31:16.456828 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:16.456918 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:16.457003 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 16:31:16.457048 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 16:31:16.461170 10929 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57034:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:31:16.467974 10929 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:31:16.496383 11389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:16.479498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:37681 (local address 127.0.0.1:50656)
0926 16:31:16.492089 (+ 12591us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:16.492131 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:16.492274 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:16.493445 (+ 1171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:16.493453 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:16.493548 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:16.494399 (+ 851us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:31:16.494404 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:16.494443 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:31:16.494530 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:31:16.495663 (+ 1133us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:31:16.495673 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:31:16.495680 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 16:31:16.495837 (+ 157us) client_negotiation.cc:240] Negotiation successful
0926 16:31:16.495938 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12322,"thread_start_us":13005,"threads_started":1}
I20260926 16:31:16.496730 11390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:16.479951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40334 (local address 127.6.33.2:36925)
0926 16:31:16.491255 (+ 11304us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:16.491268 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:16.492935 (+ 1667us) server_negotiation.cc:407] Connection header received
0926 16:31:16.493150 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:16.493157 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:16.493326 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:16.493479 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:16.495820 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:16.495828 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:16.495844 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:31:16.495871 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:16.495925 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:31:16.495944 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:16.496048 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:16.496320 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 16:31:16.496439 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11049,"thread_start_us":11316,"threads_started":1}
I20260926 16:31:16.497608 11388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:16.479613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.2:36925 (local address 127.0.0.1:40334)
0926 16:31:16.492821 (+ 13208us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:16.492840 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:16.493016 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:16.494890 (+ 1874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:16.494896 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:16.494919 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:16.495604 (+ 685us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:31:16.495607 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:16.495625 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:31:16.495667 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:31:16.496148 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:31:16.496152 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:31:16.496157 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 16:31:16.497132 (+ 975us) client_negotiation.cc:240] Negotiation successful
0926 16:31:16.497185 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13046,"thread_start_us":12044,"threads_started":1}
I20260926 16:31:16.497814 11391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:16.479925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50656 (local address 127.6.33.1:37681)
0926 16:31:16.492816 (+ 12891us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:16.492824 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:16.492861 (+ 37us) server_negotiation.cc:407] Connection header received
0926 16:31:16.492960 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:16.492970 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:16.493237 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:16.494036 (+ 799us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:16.495397 (+ 1361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:16.495409 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:16.495425 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:31:16.495451 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:16.495513 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:31:16.495540 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:16.497252 (+ 1712us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:16.497323 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 16:31:16.497418 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12699,"thread_start_us":12569,"threads_started":1}
I20260926 16:31:16.500896 11389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:16.492922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.3:33443 (local address 127.0.0.1:35126)
0926 16:31:16.496856 (+ 3934us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:16.496873 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:16.496960 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:16.499182 (+ 2222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:16.499189 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:16.499221 (+ 32us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:16.499764 (+ 543us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:31:16.499769 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:16.499787 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:31:16.499828 (+ 41us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:31:16.500441 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:31:16.500449 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:31:16.500453 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 16:31:16.500589 (+ 136us) client_negotiation.cc:240] Negotiation successful
0926 16:31:16.500638 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3816,"thread_start_us":5853,"threads_started":1}
I20260926 16:31:16.501240 11393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:16.493257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35126 (local address 127.6.33.3:33443)
0926 16:31:16.498672 (+ 5415us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:16.498682 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:16.498718 (+ 36us) server_negotiation.cc:407] Connection header received
0926 16:31:16.498784 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:16.498792 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:16.499009 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:16.499171 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:16.500196 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:16.500204 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:16.500216 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:31:16.500249 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:16.500303 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:31:16.500324 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:16.500488 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:16.500639 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 16:31:16.500721 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5258,"thread_start_us":5104,"threads_started":1}
I20260926 16:31:16.502238 11050 tablet_service.cc:1467] Processing CreateTablet for tablet 5d3eed10110a4e91aaac5bc6d8f46153 (DEFAULT_TABLE table=test-table [id=706c1d5cea264ed4b9d53d4f1c665014]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:31:16.502230 11181 tablet_service.cc:1467] Processing CreateTablet for tablet 5d3eed10110a4e91aaac5bc6d8f46153 (DEFAULT_TABLE table=test-table [id=706c1d5cea264ed4b9d53d4f1c665014]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:31:16.503629 11181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d3eed10110a4e91aaac5bc6d8f46153. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:31:16.503644 11050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d3eed10110a4e91aaac5bc6d8f46153. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:31:16.504557 11311 tablet_service.cc:1467] Processing CreateTablet for tablet 5d3eed10110a4e91aaac5bc6d8f46153 (DEFAULT_TABLE table=test-table [id=706c1d5cea264ed4b9d53d4f1c665014]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:31:16.505887 11311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d3eed10110a4e91aaac5bc6d8f46153. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:31:16.514708 11394 tablet_bootstrap.cc:492] T 5d3eed10110a4e91aaac5bc6d8f46153 P f7d7d4f67f85465ca3999dbda1830b67: Bootstrap starting.
I20260926 16:31:16.517982 11394 tablet_bootstrap.cc:654] T 5d3eed10110a4e91aaac5bc6d8f46153 P f7d7d4f67f85465ca3999dbda1830b67: Neither blocks nor log segments found. Creating new log.
I20260926 16:31:16.518172 11395 tablet_bootstrap.cc:492] T 5d3eed10110a4e91aaac5bc6d8f46153 P d46b9aafbf004df0b9810c307a502277: Bootstrap starting.
I20260926 16:31:16.519656 11396 tablet_bootstrap.cc:492] T 5d3eed10110a4e91aaac5bc6d8f46153 P 410ac670fa3846e5bb405c0f009253e4: Bootstrap starting.
I20260926 16:31:16.521000 11394 log.cc:824] T 5d3eed10110a4e91aaac5bc6d8f46153 P f7d7d4f67f85465ca3999dbda1830b67: Log is configured to *not* fsync() on all Append() calls
I20260926 16:31:16.521872 11395 tablet_bootstrap.cc:654] T 5d3eed10110a4e91aaac5bc6d8f46153 P d46b9aafbf004df0b9810c307a502277: Neither blocks nor log segments found. Creating new log.
I20260926 16:31:16.522753 11396 tablet_bootstrap.cc:654] T 5d3eed10110a4e91aaac5bc6d8f46153 P 410ac670fa3846e5bb405c0f009253e4: Neither blocks nor log segments found. Creating new log.
I20260926 16:31:16.523038 11395 log.cc:824] T 5d3eed10110a4e91aaac5bc6d8f46153 P d46b9aafbf004df0b9810c307a502277: Log is configured to *not* fsync() on all Append() calls
I20260926 16:31:16.523824 11396 log.cc:824] T 5d3eed10110a4e91aaac5bc6d8f46153 P 410ac670fa3846e5bb405c0f009253e4: Log is configured to *not* fsync() on all Append() calls
I20260926 16:31:16.524289 11394 tablet_bootstrap.cc:492] T 5d3eed10110a4e91aaac5bc6d8f46153 P f7d7d4f67f85465ca3999dbda1830b67: No bootstrap required, opened a new log
I20260926 16:31:16.524675 11394 ts_tablet_manager.cc:1397] T 5d3eed10110a4e91aaac5bc6d8f46153 P f7d7d4f67f85465ca3999dbda1830b67: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260926 16:31:16.526335 11395 tablet_bootstrap.cc:492] T 5d3eed10110a4e91aaac5bc6d8f46153 P d46b9aafbf004df0b9810c307a502277: No bootstrap required, opened a new log
I20260926 16:31:16.526626 11395 ts_tablet_manager.cc:1397] T 5d3eed10110a4e91aaac5bc6d8f46153 P d46b9aafbf004df0b9810c307a502277: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 16:31:16.532814 11396 tablet_bootstrap.cc:492] T 5d3eed10110a4e91aaac5bc6d8f46153 P 410ac670fa3846e5bb405c0f009253e4: No bootstrap required, opened a new log
I20260926 16:31:16.533108 11396 ts_tablet_manager.cc:1397] T 5d3eed10110a4e91aaac5bc6d8f46153 P 410ac670fa3846e5bb405c0f009253e4: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.009s
I20260926 16:31:16.535491 11394 raft_consensus.cc:348] T 5d3eed10110a4e91aaac5bc6d8f46153 P f7d7d4f67f85465ca3999dbda1830b67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d46b9aafbf004df0b9810c307a502277" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 37681 } } peers { permanent_uuid: "f7d7d4f67f85465ca3999dbda1830b67" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36925 } } peers { permanent_uuid: "410ac670fa3846e5bb405c0f009253e4" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 33443 } }
I20260926 16:31:16.535969 11394 raft_consensus.cc:374] T 5d3eed10110a4e91aaac5bc6d8f46153 P f7d7d4f67f85465ca3999dbda1830b67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:31:16.536126 11394 raft_consensus.cc:729] T 5d3eed10110a4e91aaac5bc6d8f46153 P f7d7d4f67f85465ca3999dbda1830b67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7d7d4f67f85465ca3999dbda1830b67, State: Initialized, Role: FOLLOWER
I20260926 16:31:16.536837 11394 consensus_queue.cc:260] T 5d3eed10110a4e91aaac5bc6d8f46153 P f7d7d4f67f85465ca3999dbda1830b67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d46b9aafbf004df0b9810c307a502277" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 37681 } } peers { permanent_uuid: "f7d7d4f67f85465ca3999dbda1830b67" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36925 } } peers { permanent_uuid: "410ac670fa3846e5bb405c0f009253e4" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 33443 } }
I20260926 16:31:16.537108 11395 raft_consensus.cc:348] T 5d3eed10110a4e91aaac5bc6d8f46153 P d46b9aafbf004df0b9810c307a502277 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d46b9aafbf004df0b9810c307a502277" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 37681 } } peers { permanent_uuid: "f7d7d4f67f85465ca3999dbda1830b67" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36925 } } peers { permanent_uuid: "410ac670fa3846e5bb405c0f009253e4" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 33443 } }
I20260926 16:31:16.537657 11395 raft_consensus.cc:374] T 5d3eed10110a4e91aaac5bc6d8f46153 P d46b9aafbf004df0b9810c307a502277 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:31:16.537982 11246 heartbeater.cc:503] Master 127.6.33.62:43931 was elected leader, sending a full tablet report...
I20260926 16:31:16.538702 11394 ts_tablet_manager.cc:1428] T 5d3eed10110a4e91aaac5bc6d8f46153 P f7d7d4f67f85465ca3999dbda1830b67: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260926 16:31:16.539621 11395 raft_consensus.cc:729] T 5d3eed10110a4e91aaac5bc6d8f46153 P d46b9aafbf004df0b9810c307a502277 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d46b9aafbf004df0b9810c307a502277, State: Initialized, Role: FOLLOWER
I20260926 16:31:16.540452 11395 consensus_queue.cc:260] T 5d3eed10110a4e91aaac5bc6d8f46153 P d46b9aafbf004df0b9810c307a502277 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d46b9aafbf004df0b9810c307a502277" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 37681 } } peers { permanent_uuid: "f7d7d4f67f85465ca3999dbda1830b67" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36925 } } peers { permanent_uuid: "410ac670fa3846e5bb405c0f009253e4" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 33443 } }
I20260926 16:31:16.541273 11396 raft_consensus.cc:348] T 5d3eed10110a4e91aaac5bc6d8f46153 P 410ac670fa3846e5bb405c0f009253e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d46b9aafbf004df0b9810c307a502277" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 37681 } } peers { permanent_uuid: "f7d7d4f67f85465ca3999dbda1830b67" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36925 } } peers { permanent_uuid: "410ac670fa3846e5bb405c0f009253e4" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 33443 } }
I20260926 16:31:16.541713 11396 raft_consensus.cc:374] T 5d3eed10110a4e91aaac5bc6d8f46153 P 410ac670fa3846e5bb405c0f009253e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:31:16.541851 11396 raft_consensus.cc:729] T 5d3eed10110a4e91aaac5bc6d8f46153 P 410ac670fa3846e5bb405c0f009253e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 410ac670fa3846e5bb405c0f009253e4, State: Initialized, Role: FOLLOWER
I20260926 16:31:16.542464 11396 consensus_queue.cc:260] T 5d3eed10110a4e91aaac5bc6d8f46153 P 410ac670fa3846e5bb405c0f009253e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d46b9aafbf004df0b9810c307a502277" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 37681 } } peers { permanent_uuid: "f7d7d4f67f85465ca3999dbda1830b67" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36925 } } peers { permanent_uuid: "410ac670fa3846e5bb405c0f009253e4" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 33443 } }
I20260926 16:31:16.547999 11376 heartbeater.cc:503] Master 127.6.33.62:43931 was elected leader, sending a full tablet report...
I20260926 16:31:16.547999 11115 heartbeater.cc:503] Master 127.6.33.62:43931 was elected leader, sending a full tablet report...
I20260926 16:31:16.548502 11396 ts_tablet_manager.cc:1428] T 5d3eed10110a4e91aaac5bc6d8f46153 P 410ac670fa3846e5bb405c0f009253e4: Time spent starting tablet: real 0.015s user 0.012s sys 0.003s
I20260926 16:31:16.549237 11395 ts_tablet_manager.cc:1428] T 5d3eed10110a4e91aaac5bc6d8f46153 P d46b9aafbf004df0b9810c307a502277: Time spent starting tablet: real 0.022s user 0.016s sys 0.004s
W20260926 16:31:16.680655 11377 tablet.cc:2369] T 5d3eed10110a4e91aaac5bc6d8f46153 P 410ac670fa3846e5bb405c0f009253e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:31:16.726004 11401 raft_consensus.cc:482] T 5d3eed10110a4e91aaac5bc6d8f46153 P d46b9aafbf004df0b9810c307a502277 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:31:16.726289 11401 raft_consensus.cc:504] T 5d3eed10110a4e91aaac5bc6d8f46153 P d46b9aafbf004df0b9810c307a502277 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d46b9aafbf004df0b9810c307a502277" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 37681 } } peers { permanent_uuid: "f7d7d4f67f85465ca3999dbda1830b67" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36925 } } peers { permanent_uuid: "410ac670fa3846e5bb405c0f009253e4" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 33443 } }
I20260926 16:31:16.727824 11401 leader_election.cc:288] T 5d3eed10110a4e91aaac5bc6d8f46153 P d46b9aafbf004df0b9810c307a502277 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f7d7d4f67f85465ca3999dbda1830b67 (127.6.33.2:36925), 410ac670fa3846e5bb405c0f009253e4 (127.6.33.3:33443)
I20260926 16:31:16.731510 11393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:16.728299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:34677 (local address 127.6.33.3:33443)
0926 16:31:16.728480 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:16.728487 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:16.729258 (+ 771us) server_negotiation.cc:407] Connection header received
0926 16:31:16.729469 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:16.729474 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:16.729539 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:16.729652 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:16.730889 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:16.730894 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:16.730898 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:31:16.730916 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:16.730949 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:31:16.730964 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:16.731086 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:16.731200 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 16:31:16.731258 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 16:31:16.731561 11390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:16.728313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:40533 (local address 127.6.33.2:36925)
0926 16:31:16.728480 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:16.728486 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:16.729257 (+ 771us) server_negotiation.cc:407] Connection header received
0926 16:31:16.729497 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:16.729503 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:16.729555 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:16.729652 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:16.730881 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:16.730887 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:16.730892 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:31:16.730912 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:16.730949 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:31:16.730964 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:16.731082 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:16.731200 (+ 118us) server_negotiation.cc:299] Negotiation successful
0926 16:31:16.731266 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 16:31:16.731863 11404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:16.728180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.2:36925 (local address 127.6.33.1:40533)
0926 16:31:16.729131 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:16.729151 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:16.729274 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:16.729696 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:16.729701 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:16.729725 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:16.730643 (+ 918us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:31:16.730649 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:16.730673 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:31:16.730722 (+ 49us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:31:16.731075 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:31:16.731080 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:31:16.731085 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 16:31:16.731257 (+ 172us) client_negotiation.cc:240] Negotiation successful
0926 16:31:16.731316 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":830,"mutex_wait_us":18,"thread_start_us":531,"threads_started":1}
I20260926 16:31:16.731879 11403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:16.728132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.3:33443 (local address 127.6.33.1:34677)
0926 16:31:16.729131 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:16.729151 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:16.729310 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:16.729696 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:16.729701 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:16.729725 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:16.730691 (+ 966us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:31:16.730694 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:16.730705 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:31:16.730730 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:31:16.731075 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:31:16.731080 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:31:16.731085 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 16:31:16.731239 (+ 154us) client_negotiation.cc:240] Negotiation successful
0926 16:31:16.731304 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":815,"spinlock_wait_cycles":34176,"thread_start_us":538,"threads_started":1}
I20260926 16:31:16.733035 11201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3eed10110a4e91aaac5bc6d8f46153" candidate_uuid: "d46b9aafbf004df0b9810c307a502277" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7d7d4f67f85465ca3999dbda1830b67" is_pre_election: true
I20260926 16:31:16.733059 11331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3eed10110a4e91aaac5bc6d8f46153" candidate_uuid: "d46b9aafbf004df0b9810c307a502277" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "410ac670fa3846e5bb405c0f009253e4" is_pre_election: true
I20260926 16:31:16.733487 11201 raft_consensus.cc:2454] T 5d3eed10110a4e91aaac5bc6d8f46153 P f7d7d4f67f85465ca3999dbda1830b67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d46b9aafbf004df0b9810c307a502277 in term 0.
I20260926 16:31:16.733515 11331 raft_consensus.cc:2454] T 5d3eed10110a4e91aaac5bc6d8f46153 P 410ac670fa3846e5bb405c0f009253e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d46b9aafbf004df0b9810c307a502277 in term 0.
I20260926 16:31:16.734102 11005 leader_election.cc:302] T 5d3eed10110a4e91aaac5bc6d8f46153 P d46b9aafbf004df0b9810c307a502277 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d46b9aafbf004df0b9810c307a502277, f7d7d4f67f85465ca3999dbda1830b67; no voters:
W20260926 16:31:16.734308 11247 tablet.cc:2369] T 5d3eed10110a4e91aaac5bc6d8f46153 P f7d7d4f67f85465ca3999dbda1830b67: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:31:16.734473 11401 raft_consensus.cc:2781] T 5d3eed10110a4e91aaac5bc6d8f46153 P d46b9aafbf004df0b9810c307a502277 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:31:16.734618 11401 raft_consensus.cc:482] T 5d3eed10110a4e91aaac5bc6d8f46153 P d46b9aafbf004df0b9810c307a502277 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:31:16.734717 11401 raft_consensus.cc:3037] T 5d3eed10110a4e91aaac5bc6d8f46153 P d46b9aafbf004df0b9810c307a502277 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:31:16.736910 11401 raft_consensus.cc:504] T 5d3eed10110a4e91aaac5bc6d8f46153 P d46b9aafbf004df0b9810c307a502277 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d46b9aafbf004df0b9810c307a502277" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 37681 } } peers { permanent_uuid: "f7d7d4f67f85465ca3999dbda1830b67" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36925 } } peers { permanent_uuid: "410ac670fa3846e5bb405c0f009253e4" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 33443 } }
I20260926 16:31:16.737555 11401 leader_election.cc:288] T 5d3eed10110a4e91aaac5bc6d8f46153 P d46b9aafbf004df0b9810c307a502277 [CANDIDATE]: Term 1 election: Requested vote from peers f7d7d4f67f85465ca3999dbda1830b67 (127.6.33.2:36925), 410ac670fa3846e5bb405c0f009253e4 (127.6.33.3:33443)
I20260926 16:31:16.738046 11201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3eed10110a4e91aaac5bc6d8f46153" candidate_uuid: "d46b9aafbf004df0b9810c307a502277" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7d7d4f67f85465ca3999dbda1830b67"
I20260926 16:31:16.738111 11331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3eed10110a4e91aaac5bc6d8f46153" candidate_uuid: "d46b9aafbf004df0b9810c307a502277" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "410ac670fa3846e5bb405c0f009253e4"
I20260926 16:31:16.738293 11201 raft_consensus.cc:3037] T 5d3eed10110a4e91aaac5bc6d8f46153 P f7d7d4f67f85465ca3999dbda1830b67 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:31:16.738322 11331 raft_consensus.cc:3037] T 5d3eed10110a4e91aaac5bc6d8f46153 P 410ac670fa3846e5bb405c0f009253e4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:31:16.741894 11201 raft_consensus.cc:2454] T 5d3eed10110a4e91aaac5bc6d8f46153 P f7d7d4f67f85465ca3999dbda1830b67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d46b9aafbf004df0b9810c307a502277 in term 1.
I20260926 16:31:16.741880 11331 raft_consensus.cc:2454] T 5d3eed10110a4e91aaac5bc6d8f46153 P 410ac670fa3846e5bb405c0f009253e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d46b9aafbf004df0b9810c307a502277 in term 1.
I20260926 16:31:16.742316 11005 leader_election.cc:302] T 5d3eed10110a4e91aaac5bc6d8f46153 P d46b9aafbf004df0b9810c307a502277 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d46b9aafbf004df0b9810c307a502277, f7d7d4f67f85465ca3999dbda1830b67; no voters:
I20260926 16:31:16.742604 11401 raft_consensus.cc:2781] T 5d3eed10110a4e91aaac5bc6d8f46153 P d46b9aafbf004df0b9810c307a502277 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:31:16.743165 11401 raft_consensus.cc:686] T 5d3eed10110a4e91aaac5bc6d8f46153 P d46b9aafbf004df0b9810c307a502277 [term 1 LEADER]: Becoming Leader. State: Replica: d46b9aafbf004df0b9810c307a502277, State: Running, Role: LEADER
I20260926 16:31:16.743677 11401 consensus_queue.cc:237] T 5d3eed10110a4e91aaac5bc6d8f46153 P d46b9aafbf004df0b9810c307a502277 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d46b9aafbf004df0b9810c307a502277" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 37681 } } peers { permanent_uuid: "f7d7d4f67f85465ca3999dbda1830b67" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36925 } } peers { permanent_uuid: "410ac670fa3846e5bb405c0f009253e4" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 33443 } }
I20260926 16:31:16.748571 10928 catalog_manager.cc:5495] T 5d3eed10110a4e91aaac5bc6d8f46153 P d46b9aafbf004df0b9810c307a502277 reported cstate change: term changed from 0 to 1, leader changed from <none> to d46b9aafbf004df0b9810c307a502277 (127.6.33.1). New cstate: current_term: 1 leader_uuid: "d46b9aafbf004df0b9810c307a502277" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d46b9aafbf004df0b9810c307a502277" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 37681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7d7d4f67f85465ca3999dbda1830b67" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "410ac670fa3846e5bb405c0f009253e4" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 33443 } health_report { overall_health: UNKNOWN } } }
I20260926 16:31:16.785593 11391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:16.783599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50666 (local address 127.6.33.1:37681)
0926 16:31:16.783780 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:16.783785 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:16.783803 (+ 18us) server_negotiation.cc:407] Connection header received
0926 16:31:16.783896 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:16.783900 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:16.783992 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:16.784079 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:16.784999 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:16.785004 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:16.785008 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:31:16.785024 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:16.785055 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:31:16.785067 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:16.785163 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:16.785316 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 16:31:16.785363 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260926 16:31:16.793416 11116 tablet.cc:2369] T 5d3eed10110a4e91aaac5bc6d8f46153 P d46b9aafbf004df0b9810c307a502277: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:31:16.794567 11201 raft_consensus.cc:1261] T 5d3eed10110a4e91aaac5bc6d8f46153 P f7d7d4f67f85465ca3999dbda1830b67 [term 1 FOLLOWER]: Refusing update from remote peer d46b9aafbf004df0b9810c307a502277: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:31:16.794566 11331 raft_consensus.cc:1261] T 5d3eed10110a4e91aaac5bc6d8f46153 P 410ac670fa3846e5bb405c0f009253e4 [term 1 FOLLOWER]: Refusing update from remote peer d46b9aafbf004df0b9810c307a502277: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:31:16.795724 11405 consensus_queue.cc:1035] T 5d3eed10110a4e91aaac5bc6d8f46153 P d46b9aafbf004df0b9810c307a502277 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7d7d4f67f85465ca3999dbda1830b67" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:31:16.796085 11401 consensus_queue.cc:1035] T 5d3eed10110a4e91aaac5bc6d8f46153 P d46b9aafbf004df0b9810c307a502277 [LEADER]: Connected to new peer: Peer: permanent_uuid: "410ac670fa3846e5bb405c0f009253e4" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 33443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:31:16.810158 11409 mvcc.cc:205] Tried to move back new op lower bound from 7333643373739163648 to 7333643373546143744. Current Snapshot: MvccSnapshot[applied={T|T < 7333643373739163648}]
I20260926 16:31:16.822100 10929 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57034:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:31:16.822403 10929 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57034:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:31:16.825995 10929 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b7bc229df4be40f9b01cfab406d2ff69: Sending DeleteTablet for 3 replicas of tablet 5d3eed10110a4e91aaac5bc6d8f46153
I20260926 16:31:16.827169 11181 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d3eed10110a4e91aaac5bc6d8f46153 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:31:16 UTC) from {username='slave'} at 127.0.0.1:40334
I20260926 16:31:16.827206 11311 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d3eed10110a4e91aaac5bc6d8f46153 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:31:16 UTC) from {username='slave'} at 127.0.0.1:35126
I20260926 16:31:16.827190 11050 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d3eed10110a4e91aaac5bc6d8f46153 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:31:16 UTC) from {username='slave'} at 127.0.0.1:50656
I20260926 16:31:16.829701 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 10988
I20260926 16:31:16.839568 11418 tablet_replica.cc:330] T 5d3eed10110a4e91aaac5bc6d8f46153 P 410ac670fa3846e5bb405c0f009253e4: stopping tablet replica
I20260926 16:31:16.841187 11418 raft_consensus.cc:2229] T 5d3eed10110a4e91aaac5bc6d8f46153 P 410ac670fa3846e5bb405c0f009253e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:31:16.843235 11418 raft_consensus.cc:2258] T 5d3eed10110a4e91aaac5bc6d8f46153 P 410ac670fa3846e5bb405c0f009253e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:31:16.845325 11419 tablet_replica.cc:330] T 5d3eed10110a4e91aaac5bc6d8f46153 P d46b9aafbf004df0b9810c307a502277: stopping tablet replica
I20260926 16:31:16.846031 11419 raft_consensus.cc:2229] T 5d3eed10110a4e91aaac5bc6d8f46153 P d46b9aafbf004df0b9810c307a502277 [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:31:16.846767 11419 raft_consensus.cc:2258] T 5d3eed10110a4e91aaac5bc6d8f46153 P d46b9aafbf004df0b9810c307a502277 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:31:16.848150 11418 ts_tablet_manager.cc:1905] T 5d3eed10110a4e91aaac5bc6d8f46153 P 410ac670fa3846e5bb405c0f009253e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:31:16.848953 11419 ts_tablet_manager.cc:1905] T 5d3eed10110a4e91aaac5bc6d8f46153 P d46b9aafbf004df0b9810c307a502277: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:31:16.850013 11391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:16.831411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50668 (local address 127.6.33.1:37681)
0926 16:31:16.831659 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:16.831669 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:16.844956 (+ 13287us) server_negotiation.cc:407] Connection header received
0926 16:31:16.845484 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:16.845490 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:16.845562 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:16.845639 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:16.849184 (+ 3545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:16.849189 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:16.849194 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:31:16.849214 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:16.849250 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:31:16.849265 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:16.849358 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:16.849635 (+ 277us) server_negotiation.cc:299] Negotiation successful
0926 16:31:16.849700 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 16:31:16.851337 11110 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:31:16.852633 11420 tablet_replica.cc:330] T 5d3eed10110a4e91aaac5bc6d8f46153 P f7d7d4f67f85465ca3999dbda1830b67: stopping tablet replica
I20260926 16:31:16.853214 11420 raft_consensus.cc:2229] T 5d3eed10110a4e91aaac5bc6d8f46153 P f7d7d4f67f85465ca3999dbda1830b67 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:31:16.853976 11420 raft_consensus.cc:2258] T 5d3eed10110a4e91aaac5bc6d8f46153 P f7d7d4f67f85465ca3999dbda1830b67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:31:16.857196 11420 ts_tablet_manager.cc:1905] T 5d3eed10110a4e91aaac5bc6d8f46153 P f7d7d4f67f85465ca3999dbda1830b67: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:31:16.859499 11418 ts_tablet_manager.cc:1918] T 5d3eed10110a4e91aaac5bc6d8f46153 P 410ac670fa3846e5bb405c0f009253e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:31:16.859675 11418 log.cc:1196] T 5d3eed10110a4e91aaac5bc6d8f46153 P 410ac670fa3846e5bb405c0f009253e4: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/ts-2/wal/wals/5d3eed10110a4e91aaac5bc6d8f46153
I20260926 16:31:16.860194 11418 ts_tablet_manager.cc:1939] T 5d3eed10110a4e91aaac5bc6d8f46153 P 410ac670fa3846e5bb405c0f009253e4: Deleting consensus metadata
I20260926 16:31:16.862160 10916 catalog_manager.cc:4841] TS 410ac670fa3846e5bb405c0f009253e4 (127.6.33.3:33443): tablet 5d3eed10110a4e91aaac5bc6d8f46153 (table test-table [id=706c1d5cea264ed4b9d53d4f1c665014]) successfully deleted
I20260926 16:31:16.862524 11420 ts_tablet_manager.cc:1918] T 5d3eed10110a4e91aaac5bc6d8f46153 P f7d7d4f67f85465ca3999dbda1830b67: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:31:16.862686 11420 log.cc:1196] T 5d3eed10110a4e91aaac5bc6d8f46153 P f7d7d4f67f85465ca3999dbda1830b67: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440233850748-6276-0/minicluster-data/ts-1/wal/wals/5d3eed10110a4e91aaac5bc6d8f46153
I20260926 16:31:16.863150 11420 ts_tablet_manager.cc:1939] T 5d3eed10110a4e91aaac5bc6d8f46153 P f7d7d4f67f85465ca3999dbda1830b67: Deleting consensus metadata
I20260926 16:31:16.864656 10915 catalog_manager.cc:4841] TS f7d7d4f67f85465ca3999dbda1830b67 (127.6.33.2:36925): tablet 5d3eed10110a4e91aaac5bc6d8f46153 (table test-table [id=706c1d5cea264ed4b9d53d4f1c665014]) successfully deleted
I20260926 16:31:16.949246 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 10988
I20260926 16:31:16.969234 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 11119
W20260926 16:31:16.969290 10916 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.33.1:37681 (error 108)
I20260926 16:31:16.970937 11389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:16.970447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:37681 (local address 127.0.0.1:50684)
0926 16:31:16.970618 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:16.970746 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.1:37681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260926 16:31:16.971164 10916 catalog_manager.cc:4584] TS d46b9aafbf004df0b9810c307a502277 (127.6.33.1:37681): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5d3eed10110a4e91aaac5bc6d8f46153: Network error: Client connection negotiation failed: client connection to 127.6.33.1:37681: connect: Connection refused (error 111)
I20260926 16:31:16.972198 11390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:16.970244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40342 (local address 127.6.33.2:36925)
0926 16:31:16.970386 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:16.970390 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:16.970406 (+ 16us) server_negotiation.cc:407] Connection header received
0926 16:31:16.970458 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:16.970462 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:16.970612 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:16.970728 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:16.971776 (+ 1048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:16.971780 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:16.971783 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:31:16.971797 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:16.971825 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:31:16.971837 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:16.971929 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:16.971965 (+ 36us) server_negotiation.cc:299] Negotiation successful
0926 16:31:16.972005 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 16:31:16.972873 11241 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:31:17.024832 11389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:17.024325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:37681 (local address 127.0.0.1:50688)
0926 16:31:17.024524 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:17.024629 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.1:37681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260926 16:31:17.025844 11389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:17.025526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:37681 (local address 127.0.0.1:50692)
0926 16:31:17.025682 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:17.025745 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.1:37681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 16:31:17.063622 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 11119
I20260926 16:31:17.079677 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 11249
I20260926 16:31:17.086500 11389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:17.086051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:37681 (local address 127.0.0.1:50706)
0926 16:31:17.086233 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:17.086319 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.1:37681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260926 16:31:17.087502 11389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:17.087143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:37681 (local address 127.0.0.1:50712)
0926 16:31:17.087285 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:17.087355 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.1:37681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 16:31:17.093329 11393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:17.080420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35134 (local address 127.6.33.3:33443)
0926 16:31:17.080631 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:17.080637 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:17.080712 (+ 75us) server_negotiation.cc:407] Connection header received
0926 16:31:17.080969 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:17.080973 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:17.081075 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:17.081178 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:17.092604 (+ 11426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:17.092609 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:17.092612 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:31:17.092632 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:17.092669 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:31:17.092681 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:17.092870 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:17.092992 (+ 122us) server_negotiation.cc:299] Negotiation successful
0926 16:31:17.093058 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 16:31:17.093995 11371 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:31:17.168042 11389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:17.167509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:37681 (local address 127.0.0.1:50720)
0926 16:31:17.167804 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:17.167886 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.1:37681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160}
I20260926 16:31:17.169193 11389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:17.168771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:37681 (local address 127.0.0.1:50724)
0926 16:31:17.168973 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:17.169058 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.1:37681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260926 16:31:17.181720 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 11249
I20260926 16:31:17.197544 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 10898
I20260926 16:31:17.198712 10959 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:31:17.302429 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 10898
2026-09-26T16:31:17Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2779 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260926 16:31:17.342859 6276 mini_oidc.cc:152] Starting JWKS server
I20260926 16:31:17.345506 6276 webserver.cc:460] Webserver started at https://127.0.0.1:45891/ using document root <none> and password file <none>
I20260926 16:31:17.345819 6276 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 16:31:17.346468 6276 webserver.cc:460] Webserver started at http://127.0.0.1:39979/ using document root <none> and password file <none>
2026-09-26T16:31:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:31:17Z Disabled control of system clock
I20260926 16:31:17.372514 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440233850748-6276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440233850748-6276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440233850748-6276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:43729
--webserver_interface=127.6.33.62
--webserver_port=0
--builtin_ntp_servers=127.6.33.20:40833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.33.62:43729
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45891/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440233850748-6276-0/testchainca.cert with env {}
W20260926 16:31:17.699147 11432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:31:17.699473 11432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:31:17.699571 11432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:31:17.706794 11432 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:31:17.706913 11432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:31:17.706954 11432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:31:17.707005 11432 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:31:17.707048 11432 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 16:31:17.707108 11432 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 16:31:17.707149 11432 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45891/jwks.json
I20260926 16:31:17.717624 11432 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:40833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440233850748-6276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440233850748-6276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.33.62:43729
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:43729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45891/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.33.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440233850748-6276-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440233850748-6276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:31:17.718932 11432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:31:17.720156 11432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:31:17.731779 11438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:17.731715 11440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:17.731992 11437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:31:17.732481 11432 server_base.cc:1034] running on GCE node
I20260926 16:31:17.733354 11432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:31:17.734828 11432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:31:17.736021 11432 hybrid_clock.cc:648] HybridClock initialized: now 1790440277735963 us; error 62 us; skew 500 ppm
I20260926 16:31:17.736420 11432 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:31:17.738796 11432 webserver.cc:460] Webserver started at http://127.6.33.62:46195/ using document root <none> and password file <none>
I20260926 16:31:17.739531 11432 fs_manager.cc:359] Metadata directory not provided
I20260926 16:31:17.739624 11432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:31:17.739957 11432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:31:17.742934 11432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440233850748-6276-0/minicluster-data/master-0/data/instance:
uuid: "3ed8d204f51240df8ee44b72081a4673"
format_stamp: "Formatted at 2026-09-26 16:31:17 on dist-test-slave-khd3"
server_key: "0cf35099f14ca07d95ea16c185d9a203"
server_key_iv: "9e98d5fc026068873ff622704cba1e94"
server_key_version: "encryptionkey@0"
I20260926 16:31:17.743731 11432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440233850748-6276-0/minicluster-data/master-0/wal/instance:
uuid: "3ed8d204f51240df8ee44b72081a4673"
format_stamp: "Formatted at 2026-09-26 16:31:17 on dist-test-slave-khd3"
server_key: "0cf35099f14ca07d95ea16c185d9a203"
server_key_iv: "9e98d5fc026068873ff622704cba1e94"
server_key_version: "encryptionkey@0"
I20260926 16:31:17.748970 11432 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260926 16:31:17.752525 11446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:17.754076 11432 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:31:17.754273 11432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440233850748-6276-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440233850748-6276-0/minicluster-data/master-0/wal
uuid: "3ed8d204f51240df8ee44b72081a4673"
format_stamp: "Formatted at 2026-09-26 16:31:17 on dist-test-slave-khd3"
server_key: "0cf35099f14ca07d95ea16c185d9a203"
server_key_iv: "9e98d5fc026068873ff622704cba1e94"
server_key_version: "encryptionkey@0"
I20260926 16:31:17.754498 11432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440233850748-6276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440233850748-6276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440233850748-6276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:31:17.771540 11432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:31:17.772418 11432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:31:17.772730 11432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:31:17.791366 11432 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.62:43729
I20260926 16:31:17.791385 11497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.62:43729 every 8 connection(s)
I20260926 16:31:17.792779 11432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
I20260926 16:31:17.794426 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 11432
I20260926 16:31:17.794620 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440233850748-6276-0/minicluster-data/master-0/wal/instance
I20260926 16:31:17.795419 6276 external_mini_cluster.cc:1428] Setting key 26d97ab3db668a57bfc03cebaff38829
I20260926 16:31:17.798239 11498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:31:17.808797 11498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ed8d204f51240df8ee44b72081a4673: Bootstrap starting.
I20260926 16:31:17.808915 11501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:17.797460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38894 (local address 127.6.33.62:43729)
0926 16:31:17.798487 (+ 1027us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:17.798506 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:17.798553 (+ 47us) server_negotiation.cc:407] Connection header received
0926 16:31:17.800151 (+ 1598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:17.800172 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:17.800658 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:17.801140 (+ 482us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:17.801720 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:17.803048 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:17.804375 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:17.805110 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:17.805305 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:17.806543 (+ 1238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:17.806551 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:17.806568 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:31:17.807150 (+ 582us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:17.807220 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:31:17.807244 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:17.807366 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:17.807584 (+ 218us) server_negotiation.cc:299] Negotiation successful
0926 16:31:17.807910 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":314,"threads_started":1}
I20260926 16:31:17.812659 11498 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ed8d204f51240df8ee44b72081a4673: Neither blocks nor log segments found. Creating new log.
I20260926 16:31:17.812821 6276 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:31:17.813735 11498 log.cc:824] T 00000000000000000000000000000000 P 3ed8d204f51240df8ee44b72081a4673: Log is configured to *not* fsync() on all Append() calls
I20260926 16:31:17.816833 11498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ed8d204f51240df8ee44b72081a4673: No bootstrap required, opened a new log
I20260926 16:31:17.824527 11498 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ed8d204f51240df8ee44b72081a4673 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ed8d204f51240df8ee44b72081a4673" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 43729 } }
I20260926 16:31:17.824980 11498 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ed8d204f51240df8ee44b72081a4673 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:31:17.825098 11498 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ed8d204f51240df8ee44b72081a4673 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ed8d204f51240df8ee44b72081a4673, State: Initialized, Role: FOLLOWER
I20260926 16:31:17.825671 11498 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ed8d204f51240df8ee44b72081a4673 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ed8d204f51240df8ee44b72081a4673" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 43729 } }
I20260926 16:31:17.825908 11498 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ed8d204f51240df8ee44b72081a4673 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:31:17.826050 11498 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ed8d204f51240df8ee44b72081a4673 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:31:17.826234 11498 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ed8d204f51240df8ee44b72081a4673 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:31:17.828418 11498 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ed8d204f51240df8ee44b72081a4673 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ed8d204f51240df8ee44b72081a4673" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 43729 } }
I20260926 16:31:17.829041 11498 leader_election.cc:302] T 00000000000000000000000000000000 P 3ed8d204f51240df8ee44b72081a4673 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ed8d204f51240df8ee44b72081a4673; no voters:
I20260926 16:31:17.829700 11498 leader_election.cc:288] T 00000000000000000000000000000000 P 3ed8d204f51240df8ee44b72081a4673 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:31:17.829846 11503 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ed8d204f51240df8ee44b72081a4673 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:31:17.831022 11503 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ed8d204f51240df8ee44b72081a4673 [term 1 LEADER]: Becoming Leader. State: Replica: 3ed8d204f51240df8ee44b72081a4673, State: Running, Role: LEADER
I20260926 16:31:17.831516 11503 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ed8d204f51240df8ee44b72081a4673 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ed8d204f51240df8ee44b72081a4673" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 43729 } }
I20260926 16:31:17.831868 11498 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ed8d204f51240df8ee44b72081a4673 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:31:17.834347 11504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ed8d204f51240df8ee44b72081a4673 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ed8d204f51240df8ee44b72081a4673" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ed8d204f51240df8ee44b72081a4673" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 43729 } } }
I20260926 16:31:17.834437 11505 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ed8d204f51240df8ee44b72081a4673 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ed8d204f51240df8ee44b72081a4673. Latest consensus state: current_term: 1 leader_uuid: "3ed8d204f51240df8ee44b72081a4673" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ed8d204f51240df8ee44b72081a4673" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 43729 } } }
I20260926 16:31:17.834769 11504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ed8d204f51240df8ee44b72081a4673 [sys.catalog]: This master's current role is: LEADER
I20260926 16:31:17.834821 11505 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ed8d204f51240df8ee44b72081a4673 [sys.catalog]: This master's current role is: LEADER
I20260926 16:31:17.835666 11509 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:31:17.842402 11509 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:31:17.850912 11509 catalog_manager.cc:1347] Generated new cluster ID: 8c18b086252b4c5eb3b201f77894380b
I20260926 16:31:17.851046 11509 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:31:17.867076 11509 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:31:17.868984 11509 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:31:17.876528 11509 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ed8d204f51240df8ee44b72081a4673: Generated new TSK 0
I20260926 16:31:17.877589 11509 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 16:31:17.916702 11528 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260926 16:31:17.916918 11528 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 16:31:17.917841 11501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:17.896653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38904 (local address 127.6.33.62:43729)
0926 16:31:17.896923 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:17.896930 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:17.897344 (+ 414us) server_negotiation.cc:407] Connection header received
0926 16:31:17.897732 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:17.897736 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:17.897812 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:17.897955 (+ 143us) server_negotiation.cc:226] Negotiated authn=JWT
0926 16:31:17.898554 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:17.899373 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:17.901405 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:17.901898 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:17.902016 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:17.902628 (+ 612us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 16:31:17.917309 (+ 14681us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45891/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 16:31:17.917610 (+ 301us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38904: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45891/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":66}
W20260926 16:31:17.917984 11501 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38904: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45891/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260926 16:31:17.919760 11527 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:31:17.896503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:43729 (local address 127.0.0.1:38904)
0926 16:31:17.897216 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:17.897237 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:17.897428 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:17.897994 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:17.897999 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:17.898012 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0926 16:31:17.898386 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:17.898399 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:17.899522 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:17.899528 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:17.901263 (+ 1735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:17.901279 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:17.902036 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:17.902083 (+ 47us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:17.902287 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:17.902364 (+ 77us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 16:31:17.917615 (+ 15251us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45891/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 16:31:17.917974 (+ 359us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.33.62:43729: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45891/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":384,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.6.33.62:43729: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45891/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260926 16:31:17.921604 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 11432
I20260926 16:31:17.922667 11493 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:31:18.016265 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 11432
2026-09-26T16:31:18Z chronyd exiting
I20260926 16:31:18.066228 6276 test_util.cc:175] -----------------------------------------------
I20260926 16:31:18.066349 6276 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440233850748-6276-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (727 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260926 16:31:18.070035 6276 mini_oidc.cc:152] Starting JWKS server
I20260926 16:31:18.071750 6276 webserver.cc:460] Webserver started at https://127.0.0.1:36229/ using document root <none> and password file <none>
I20260926 16:31:18.071949 6276 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 16:31:18.072511 6276 webserver.cc:460] Webserver started at http://127.0.0.1:37155/ using document root <none> and password file <none>
2026-09-26T16:31:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:31:18Z Disabled control of system clock
I20260926 16:31:18.097908 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440233850748-6276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440233850748-6276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440233850748-6276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:43937
--webserver_interface=127.6.33.62
--webserver_port=0
--builtin_ntp_servers=127.6.33.20:34345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.33.62:43937
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36229/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440233850748-6276-0/testchainca.cert with env {}
W20260926 16:31:18.419116 11535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:31:18.419420 11535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:31:18.419507 11535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:31:18.427050 11535 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:31:18.427174 11535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:31:18.427212 11535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:31:18.427271 11535 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:31:18.427315 11535 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 16:31:18.427371 11535 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 16:31:18.427410 11535 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36229/jwks.json
I20260926 16:31:18.437669 11535 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:34345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440233850748-6276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440233850748-6276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.33.62:43937
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:43937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36229/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.33.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440233850748-6276-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440233850748-6276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:31:18.439021 11535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:31:18.440331 11535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:31:18.452376 11540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:18.452400 11543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:18.452378 11541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:31:18.453327 11535 server_base.cc:1034] running on GCE node
I20260926 16:31:18.454188 11535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:31:18.455595 11535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:31:18.456868 11535 hybrid_clock.cc:648] HybridClock initialized: now 1790440278456808 us; error 42 us; skew 500 ppm
I20260926 16:31:18.457238 11535 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:31:18.459559 11535 webserver.cc:460] Webserver started at http://127.6.33.62:39203/ using document root <none> and password file <none>
I20260926 16:31:18.460259 11535 fs_manager.cc:359] Metadata directory not provided
I20260926 16:31:18.460392 11535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:31:18.460714 11535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:31:18.463615 11535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440233850748-6276-0/minicluster-data/master-0/data/instance:
uuid: "c2c7c1cfc96b4a408df65d5a5dfb06d9"
format_stamp: "Formatted at 2026-09-26 16:31:18 on dist-test-slave-khd3"
server_key: "b62183f1300eaef9521c5b20f1e75158"
server_key_iv: "cfa8cdfdd3a13b148f551fbc9e99093a"
server_key_version: "encryptionkey@0"
I20260926 16:31:18.464397 11535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440233850748-6276-0/minicluster-data/master-0/wal/instance:
uuid: "c2c7c1cfc96b4a408df65d5a5dfb06d9"
format_stamp: "Formatted at 2026-09-26 16:31:18 on dist-test-slave-khd3"
server_key: "b62183f1300eaef9521c5b20f1e75158"
server_key_iv: "cfa8cdfdd3a13b148f551fbc9e99093a"
server_key_version: "encryptionkey@0"
I20260926 16:31:18.469650 11535 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 16:31:18.473181 11549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:18.474561 11535 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260926 16:31:18.474747 11535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440233850748-6276-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440233850748-6276-0/minicluster-data/master-0/wal
uuid: "c2c7c1cfc96b4a408df65d5a5dfb06d9"
format_stamp: "Formatted at 2026-09-26 16:31:18 on dist-test-slave-khd3"
server_key: "b62183f1300eaef9521c5b20f1e75158"
server_key_iv: "cfa8cdfdd3a13b148f551fbc9e99093a"
server_key_version: "encryptionkey@0"
I20260926 16:31:18.474957 11535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440233850748-6276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440233850748-6276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440233850748-6276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:31:18.489238 11535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:31:18.490108 11535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:31:18.490412 11535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:31:18.510052 11535 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.62:43937
I20260926 16:31:18.510094 11600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.62:43937 every 8 connection(s)
I20260926 16:31:18.511518 11535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
I20260926 16:31:18.515341 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 11535
I20260926 16:31:18.515548 11601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:31:18.515645 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440233850748-6276-0/minicluster-data/master-0/wal/instance
I20260926 16:31:18.516309 6276 external_mini_cluster.cc:1428] Setting key 9c0ba9db1a2484d37836710adbcd7b72
I20260926 16:31:18.530707 11601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2c7c1cfc96b4a408df65d5a5dfb06d9: Bootstrap starting.
I20260926 16:31:18.532938 11604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:18.518533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44816 (local address 127.6.33.62:43937)
0926 16:31:18.519724 (+ 1191us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:18.519740 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:18.519795 (+ 55us) server_negotiation.cc:407] Connection header received
0926 16:31:18.522595 (+ 2800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:18.522626 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:18.523313 (+ 687us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:18.523800 (+ 487us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:18.524342 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:18.526215 (+ 1873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:18.527542 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:18.528661 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:18.528880 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:18.530047 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:18.530067 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:18.530089 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:31:18.530819 (+ 730us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:18.530927 (+ 108us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:31:18.530953 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:18.531082 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:18.531251 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 16:31:18.531591 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":329,"threads_started":1}
I20260926 16:31:18.534574 11601 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2c7c1cfc96b4a408df65d5a5dfb06d9: Neither blocks nor log segments found. Creating new log.
I20260926 16:31:18.535831 11601 log.cc:824] T 00000000000000000000000000000000 P c2c7c1cfc96b4a408df65d5a5dfb06d9: Log is configured to *not* fsync() on all Append() calls
I20260926 16:31:18.537187 6276 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:31:18.538841 11601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2c7c1cfc96b4a408df65d5a5dfb06d9: No bootstrap required, opened a new log
I20260926 16:31:18.546433 11601 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2c7c1cfc96b4a408df65d5a5dfb06d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2c7c1cfc96b4a408df65d5a5dfb06d9" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 43937 } }
I20260926 16:31:18.546814 11601 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2c7c1cfc96b4a408df65d5a5dfb06d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:31:18.546949 11601 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2c7c1cfc96b4a408df65d5a5dfb06d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2c7c1cfc96b4a408df65d5a5dfb06d9, State: Initialized, Role: FOLLOWER
I20260926 16:31:18.547590 11601 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2c7c1cfc96b4a408df65d5a5dfb06d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2c7c1cfc96b4a408df65d5a5dfb06d9" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 43937 } }
I20260926 16:31:18.547801 11601 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2c7c1cfc96b4a408df65d5a5dfb06d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:31:18.547962 11601 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2c7c1cfc96b4a408df65d5a5dfb06d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:31:18.548152 11601 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2c7c1cfc96b4a408df65d5a5dfb06d9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:31:18.550513 11601 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2c7c1cfc96b4a408df65d5a5dfb06d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2c7c1cfc96b4a408df65d5a5dfb06d9" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 43937 } }
I20260926 16:31:18.551160 11601 leader_election.cc:302] T 00000000000000000000000000000000 P c2c7c1cfc96b4a408df65d5a5dfb06d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2c7c1cfc96b4a408df65d5a5dfb06d9; no voters:
I20260926 16:31:18.551784 11601 leader_election.cc:288] T 00000000000000000000000000000000 P c2c7c1cfc96b4a408df65d5a5dfb06d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:31:18.551976 11606 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2c7c1cfc96b4a408df65d5a5dfb06d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:31:18.552685 11606 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2c7c1cfc96b4a408df65d5a5dfb06d9 [term 1 LEADER]: Becoming Leader. State: Replica: c2c7c1cfc96b4a408df65d5a5dfb06d9, State: Running, Role: LEADER
I20260926 16:31:18.553308 11606 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2c7c1cfc96b4a408df65d5a5dfb06d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2c7c1cfc96b4a408df65d5a5dfb06d9" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 43937 } }
I20260926 16:31:18.553962 11601 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2c7c1cfc96b4a408df65d5a5dfb06d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:31:18.556610 11608 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2c7c1cfc96b4a408df65d5a5dfb06d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2c7c1cfc96b4a408df65d5a5dfb06d9. Latest consensus state: current_term: 1 leader_uuid: "c2c7c1cfc96b4a408df65d5a5dfb06d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2c7c1cfc96b4a408df65d5a5dfb06d9" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 43937 } } }
I20260926 16:31:18.557060 11608 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2c7c1cfc96b4a408df65d5a5dfb06d9 [sys.catalog]: This master's current role is: LEADER
I20260926 16:31:18.556975 11607 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2c7c1cfc96b4a408df65d5a5dfb06d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2c7c1cfc96b4a408df65d5a5dfb06d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2c7c1cfc96b4a408df65d5a5dfb06d9" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 43937 } } }
I20260926 16:31:18.557278 11607 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2c7c1cfc96b4a408df65d5a5dfb06d9 [sys.catalog]: This master's current role is: LEADER
W20260926 16:31:18.565948 11622 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c2c7c1cfc96b4a408df65d5a5dfb06d9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 16:31:18.566105 11622 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 16:31:18.566246 11613 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:31:18.567914 11613 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:31:18.575840 11613 catalog_manager.cc:1347] Generated new cluster ID: b470c7bfe9e846dd9d13fd786d45318e
I20260926 16:31:18.575961 11613 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:31:18.590236 11613 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:31:18.591534 11613 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:31:18.598642 11613 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2c7c1cfc96b4a408df65d5a5dfb06d9: Generated new TSK 0
I20260926 16:31:18.599481 11613 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 16:31:18.644809 11631 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260926 16:31:18.644951 11631 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 16:31:18.646134 11604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:18.618670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44830 (local address 127.6.33.62:43937)
0926 16:31:18.618873 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:18.618882 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:18.619288 (+ 406us) server_negotiation.cc:407] Connection header received
0926 16:31:18.619550 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:18.619554 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:18.619625 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:18.619774 (+ 149us) server_negotiation.cc:226] Negotiated authn=JWT
0926 16:31:18.620207 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:18.621009 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:18.622450 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:18.622860 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:18.622987 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:18.623476 (+ 489us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 16:31:18.645351 (+ 21875us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36229/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 16:31:18.645775 (+ 424us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44830: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36229/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":53}
W20260926 16:31:18.646387 11604 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44830: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36229/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260926 16:31:18.646512 11630 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:31:18.618589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:43937 (local address 127.0.0.1:44830)
0926 16:31:18.619180 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:18.619199 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:18.619362 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:18.619786 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:18.619791 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:18.619802 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0926 16:31:18.620066 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:18.620074 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:18.621147 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:18.621155 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:18.622298 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:18.622309 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:18.622975 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:18.623041 (+ 66us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:18.623187 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:18.623218 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 16:31:18.645615 (+ 22397us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36229/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 16:31:18.645908 (+ 293us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.33.62:43937: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36229/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":343,"threads_started":1}
I20260926 16:31:18.648236 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 11535
I20260926 16:31:18.649472 11596 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:31:18.743229 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 11535
2026-09-26T16:31:18Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (728 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260926 16:31:18.798749 6276 mini_oidc.cc:152] Starting JWKS server
I20260926 16:31:18.800576 6276 webserver.cc:460] Webserver started at https://127.0.0.1:46193/ using document root <none> and password file <none>
I20260926 16:31:18.800855 6276 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 16:31:18.801461 6276 webserver.cc:460] Webserver started at http://127.0.0.1:38391/ using document root <none> and password file <none>
2026-09-26T16:31:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:31:18Z Disabled control of system clock
I20260926 16:31:18.828081 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790440233850748-6276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790440233850748-6276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790440233850748-6276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:35417
--webserver_interface=127.6.33.62
--webserver_port=0
--builtin_ntp_servers=127.6.33.20:36623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.33.62:35417
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46193/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 16:31:19.154646 11638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:31:19.155054 11638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:31:19.155236 11638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:31:19.162637 11638 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:31:19.162760 11638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:31:19.162842 11638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:31:19.162925 11638 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:31:19.163010 11638 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 16:31:19.163112 11638 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 16:31:19.163164 11638 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46193/jwks.json
I20260926 16:31:19.172431 11638 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:36623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790440233850748-6276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790440233850748-6276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.33.62:35417
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:35417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46193/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790440233850748-6276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:31:19.173828 11638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:31:19.175659 11638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:31:19.187675 11644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:19.187636 11643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:19.187711 11646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:31:19.188427 11638 server_base.cc:1034] running on GCE node
I20260926 16:31:19.189338 11638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:31:19.190845 11638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:31:19.192049 11638 hybrid_clock.cc:648] HybridClock initialized: now 1790440279192004 us; error 41 us; skew 500 ppm
I20260926 16:31:19.192411 11638 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:31:19.194852 11638 webserver.cc:460] Webserver started at http://127.6.33.62:40991/ using document root <none> and password file <none>
I20260926 16:31:19.195595 11638 fs_manager.cc:359] Metadata directory not provided
I20260926 16:31:19.195681 11638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:31:19.196023 11638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:31:19.199131 11638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790440233850748-6276-0/minicluster-data/master-0/data/instance:
uuid: "cd1422ee62e84b9b829c5a2c9e602f58"
format_stamp: "Formatted at 2026-09-26 16:31:19 on dist-test-slave-khd3"
server_key: "5224e599dfeb5444b307f28bd49be69f"
server_key_iv: "eb00e72ecf4cb245e15311f553102b0a"
server_key_version: "encryptionkey@0"
I20260926 16:31:19.199996 11638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790440233850748-6276-0/minicluster-data/master-0/wal/instance:
uuid: "cd1422ee62e84b9b829c5a2c9e602f58"
format_stamp: "Formatted at 2026-09-26 16:31:19 on dist-test-slave-khd3"
server_key: "5224e599dfeb5444b307f28bd49be69f"
server_key_iv: "eb00e72ecf4cb245e15311f553102b0a"
server_key_version: "encryptionkey@0"
I20260926 16:31:19.205765 11638 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260926 16:31:19.209421 11652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:19.210877 11638 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:31:19.211071 11638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790440233850748-6276-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790440233850748-6276-0/minicluster-data/master-0/wal
uuid: "cd1422ee62e84b9b829c5a2c9e602f58"
format_stamp: "Formatted at 2026-09-26 16:31:19 on dist-test-slave-khd3"
server_key: "5224e599dfeb5444b307f28bd49be69f"
server_key_iv: "eb00e72ecf4cb245e15311f553102b0a"
server_key_version: "encryptionkey@0"
I20260926 16:31:19.211301 11638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790440233850748-6276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790440233850748-6276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790440233850748-6276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:31:19.235369 11638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:31:19.236246 11638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:31:19.236536 11638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:31:19.254734 11638 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.62:35417
I20260926 16:31:19.254752 11703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.62:35417 every 8 connection(s)
I20260926 16:31:19.256134 11638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
I20260926 16:31:19.259995 11704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:31:19.266213 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 11638
I20260926 16:31:19.266422 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790440233850748-6276-0/minicluster-data/master-0/wal/instance
I20260926 16:31:19.267470 6276 external_mini_cluster.cc:1428] Setting key 780ecfb3f5c17e6e992dd8a1feb1ccb5
I20260926 16:31:19.272202 11704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd1422ee62e84b9b829c5a2c9e602f58: Bootstrap starting.
I20260926 16:31:19.275285 11704 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd1422ee62e84b9b829c5a2c9e602f58: Neither blocks nor log segments found. Creating new log.
I20260926 16:31:19.276288 11704 log.cc:824] T 00000000000000000000000000000000 P cd1422ee62e84b9b829c5a2c9e602f58: Log is configured to *not* fsync() on all Append() calls
I20260926 16:31:19.279177 11704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd1422ee62e84b9b829c5a2c9e602f58: No bootstrap required, opened a new log
I20260926 16:31:19.281042 11707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:19.269621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52286 (local address 127.6.33.62:35417)
0926 16:31:19.270755 (+ 1134us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:19.270774 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:19.270816 (+ 42us) server_negotiation.cc:407] Connection header received
0926 16:31:19.272465 (+ 1649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:19.272488 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:19.272988 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:19.273353 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:19.273895 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:19.275114 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:19.276426 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:19.277179 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:19.277382 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:19.278487 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:19.278494 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:19.278508 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:31:19.279010 (+ 502us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:19.279082 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:31:19.279104 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:19.279229 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:19.279462 (+ 233us) server_negotiation.cc:299] Negotiation successful
0926 16:31:19.279804 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":343,"threads_started":1}
I20260926 16:31:19.285132 6276 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:31:19.287871 11704 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd1422ee62e84b9b829c5a2c9e602f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd1422ee62e84b9b829c5a2c9e602f58" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 35417 } }
I20260926 16:31:19.288210 11704 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd1422ee62e84b9b829c5a2c9e602f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:31:19.288328 11704 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd1422ee62e84b9b829c5a2c9e602f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd1422ee62e84b9b829c5a2c9e602f58, State: Initialized, Role: FOLLOWER
I20260926 16:31:19.288964 11704 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd1422ee62e84b9b829c5a2c9e602f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd1422ee62e84b9b829c5a2c9e602f58" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 35417 } }
I20260926 16:31:19.289167 11704 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd1422ee62e84b9b829c5a2c9e602f58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:31:19.289266 11704 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd1422ee62e84b9b829c5a2c9e602f58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:31:19.289384 11704 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd1422ee62e84b9b829c5a2c9e602f58 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:31:19.291483 11704 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd1422ee62e84b9b829c5a2c9e602f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd1422ee62e84b9b829c5a2c9e602f58" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 35417 } }
I20260926 16:31:19.291992 11704 leader_election.cc:302] T 00000000000000000000000000000000 P cd1422ee62e84b9b829c5a2c9e602f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd1422ee62e84b9b829c5a2c9e602f58; no voters:
I20260926 16:31:19.292696 11704 leader_election.cc:288] T 00000000000000000000000000000000 P cd1422ee62e84b9b829c5a2c9e602f58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:31:19.292850 11709 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd1422ee62e84b9b829c5a2c9e602f58 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:31:19.293644 11709 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd1422ee62e84b9b829c5a2c9e602f58 [term 1 LEADER]: Becoming Leader. State: Replica: cd1422ee62e84b9b829c5a2c9e602f58, State: Running, Role: LEADER
I20260926 16:31:19.294204 11709 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd1422ee62e84b9b829c5a2c9e602f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd1422ee62e84b9b829c5a2c9e602f58" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 35417 } }
I20260926 16:31:19.295056 11704 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd1422ee62e84b9b829c5a2c9e602f58 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:31:19.297734 11710 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd1422ee62e84b9b829c5a2c9e602f58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd1422ee62e84b9b829c5a2c9e602f58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd1422ee62e84b9b829c5a2c9e602f58" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 35417 } } }
I20260926 16:31:19.298087 11710 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd1422ee62e84b9b829c5a2c9e602f58 [sys.catalog]: This master's current role is: LEADER
I20260926 16:31:19.297920 11711 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd1422ee62e84b9b829c5a2c9e602f58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd1422ee62e84b9b829c5a2c9e602f58. Latest consensus state: current_term: 1 leader_uuid: "cd1422ee62e84b9b829c5a2c9e602f58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd1422ee62e84b9b829c5a2c9e602f58" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 35417 } } }
I20260926 16:31:19.298328 11711 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd1422ee62e84b9b829c5a2c9e602f58 [sys.catalog]: This master's current role is: LEADER
I20260926 16:31:19.301646 11715 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:31:19.306756 11715 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:31:19.315591 11715 catalog_manager.cc:1347] Generated new cluster ID: 9f648af600734502a3eed51560f91042
I20260926 16:31:19.315764 11715 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:31:19.329609 11715 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:31:19.331604 11715 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:31:19.355508 11715 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd1422ee62e84b9b829c5a2c9e602f58: Generated new TSK 0
I20260926 16:31:19.356841 11715 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:31:19.385828 11707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:19.377718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52290 (local address 127.6.33.62:35417)
0926 16:31:19.377907 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:19.377915 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:19.378341 (+ 426us) server_negotiation.cc:407] Connection header received
0926 16:31:19.378556 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:19.378562 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:19.378668 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:19.378840 (+ 172us) server_negotiation.cc:226] Negotiated authn=JWT
0926 16:31:19.379235 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:19.380359 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:19.381799 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:19.382249 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:19.382379 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:19.382895 (+ 516us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 16:31:19.385213 (+ 2318us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46193/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 16:31:19.385521 (+ 308us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52290: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46193/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":66}
W20260926 16:31:19.385952 11707 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52290: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46193/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260926 16:31:19.387987 11733 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:31:19.377612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:35417 (local address 127.0.0.1:52290)
0926 16:31:19.378241 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:19.378257 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:19.378388 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:19.378842 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:19.378846 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:19.378855 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0926 16:31:19.379099 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:19.379107 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:19.380520 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:19.380524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:19.381644 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:19.381656 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:19.382369 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:19.382444 (+ 75us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:19.382580 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:19.382612 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 16:31:19.385424 (+ 2812us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46193/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 16:31:19.387507 (+ 2083us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.33.62:35417: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46193/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":501,"mutex_wait_us":1774,"thread_start_us":353,"threads_started":1}
I20260926 16:31:19.389285 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 11638
I20260926 16:31:19.390506 11699 generic_service.cc:196] Checking for leaks (request via RPC)
E20260926 16:31:19.397835 11734 webserver.cc:533] Webserver: sslize failed:
E20260926 16:31:19.397928 11734 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 16:31:19.480693 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 11638
2026-09-26T16:31:19Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (737 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-26T16:31:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:31:19Z Disabled control of system clock
I20260926 16:31:19.562909 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440233850748-6276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440233850748-6276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440233850748-6276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:43365
--webserver_interface=127.6.33.62
--webserver_port=0
--builtin_ntp_servers=127.6.33.20:34641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.33.62:43365
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440233850748-6276-0/testchainca.cert with env {}
W20260926 16:31:19.887604 11739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:31:19.887902 11739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:31:19.887984 11739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:31:19.895116 11739 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:31:19.895216 11739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:31:19.895254 11739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:31:19.895311 11739 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:31:19.895354 11739 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 16:31:19.895416 11739 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 16:31:19.895455 11739 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260926 16:31:19.905259 11739 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:34641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440233850748-6276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440233850748-6276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.33.62:43365
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:43365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.33.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440233850748-6276-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440233850748-6276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:31:19.906533 11739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:31:19.908008 11739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:31:19.919905 11744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:19.919889 11745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:19.919950 11747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:31:19.920848 11739 server_base.cc:1034] running on GCE node
I20260926 16:31:19.921666 11739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:31:19.923126 11739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:31:19.924362 11739 hybrid_clock.cc:648] HybridClock initialized: now 1790440279924277 us; error 79 us; skew 500 ppm
I20260926 16:31:19.924726 11739 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:31:19.927111 11739 webserver.cc:460] Webserver started at http://127.6.33.62:37207/ using document root <none> and password file <none>
I20260926 16:31:19.927800 11739 fs_manager.cc:359] Metadata directory not provided
I20260926 16:31:19.927932 11739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:31:19.928244 11739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:31:19.931118 11739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440233850748-6276-0/minicluster-data/master-0/data/instance:
uuid: "034e3933264148c3a9b8b6b110c7a4c4"
format_stamp: "Formatted at 2026-09-26 16:31:19 on dist-test-slave-khd3"
server_key: "653ca6d0b1335807fec7c353b16cad51"
server_key_iv: "5c3ad37357276ab16bea2835b212f47b"
server_key_version: "encryptionkey@0"
I20260926 16:31:19.931878 11739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440233850748-6276-0/minicluster-data/master-0/wal/instance:
uuid: "034e3933264148c3a9b8b6b110c7a4c4"
format_stamp: "Formatted at 2026-09-26 16:31:19 on dist-test-slave-khd3"
server_key: "653ca6d0b1335807fec7c353b16cad51"
server_key_iv: "5c3ad37357276ab16bea2835b212f47b"
server_key_version: "encryptionkey@0"
I20260926 16:31:19.936959 11739 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260926 16:31:19.940344 11753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:19.941825 11739 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:31:19.942021 11739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440233850748-6276-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440233850748-6276-0/minicluster-data/master-0/wal
uuid: "034e3933264148c3a9b8b6b110c7a4c4"
format_stamp: "Formatted at 2026-09-26 16:31:19 on dist-test-slave-khd3"
server_key: "653ca6d0b1335807fec7c353b16cad51"
server_key_iv: "5c3ad37357276ab16bea2835b212f47b"
server_key_version: "encryptionkey@0"
I20260926 16:31:19.942238 11739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440233850748-6276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440233850748-6276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440233850748-6276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:31:19.963038 11739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:31:19.963943 11739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:31:19.964251 11739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:31:19.982870 11739 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.62:43365
I20260926 16:31:19.982916 11804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.62:43365 every 8 connection(s)
I20260926 16:31:19.984241 11739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
I20260926 16:31:19.988274 11805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:31:19.992255 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 11739
I20260926 16:31:19.992483 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440233850748-6276-0/minicluster-data/master-0/wal/instance
I20260926 16:31:19.993284 6276 external_mini_cluster.cc:1428] Setting key 4f168cfa9b19722dd4ede9799b46877b
I20260926 16:31:20.001714 11805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 034e3933264148c3a9b8b6b110c7a4c4: Bootstrap starting.
I20260926 16:31:20.004834 11805 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 034e3933264148c3a9b8b6b110c7a4c4: Neither blocks nor log segments found. Creating new log.
I20260926 16:31:20.006003 11805 log.cc:824] T 00000000000000000000000000000000 P 034e3933264148c3a9b8b6b110c7a4c4: Log is configured to *not* fsync() on all Append() calls
I20260926 16:31:20.007134 11808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:19.995440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57986 (local address 127.6.33.62:43365)
0926 16:31:19.996462 (+ 1022us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:19.996478 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:19.996524 (+ 46us) server_negotiation.cc:407] Connection header received
0926 16:31:19.998226 (+ 1702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:19.998248 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:19.998772 (+ 524us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:19.999215 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:19.999830 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:20.001134 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:20.002445 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:20.003123 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:20.003312 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:20.004440 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:20.004448 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:20.004463 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:31:20.005094 (+ 631us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:20.005193 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:31:20.005216 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:20.005347 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:20.005572 (+ 225us) server_negotiation.cc:299] Negotiation successful
0926 16:31:20.005880 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":326,"threads_started":1}
I20260926 16:31:20.009282 11805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 034e3933264148c3a9b8b6b110c7a4c4: No bootstrap required, opened a new log
I20260926 16:31:20.011381 6276 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:31:20.017827 11805 raft_consensus.cc:348] T 00000000000000000000000000000000 P 034e3933264148c3a9b8b6b110c7a4c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "034e3933264148c3a9b8b6b110c7a4c4" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 43365 } }
I20260926 16:31:20.018241 11805 raft_consensus.cc:374] T 00000000000000000000000000000000 P 034e3933264148c3a9b8b6b110c7a4c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:31:20.018350 11805 raft_consensus.cc:729] T 00000000000000000000000000000000 P 034e3933264148c3a9b8b6b110c7a4c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 034e3933264148c3a9b8b6b110c7a4c4, State: Initialized, Role: FOLLOWER
I20260926 16:31:20.019030 11805 consensus_queue.cc:260] T 00000000000000000000000000000000 P 034e3933264148c3a9b8b6b110c7a4c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "034e3933264148c3a9b8b6b110c7a4c4" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 43365 } }
I20260926 16:31:20.019286 11805 raft_consensus.cc:388] T 00000000000000000000000000000000 P 034e3933264148c3a9b8b6b110c7a4c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:31:20.019392 11805 raft_consensus.cc:482] T 00000000000000000000000000000000 P 034e3933264148c3a9b8b6b110c7a4c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:31:20.019527 11805 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 034e3933264148c3a9b8b6b110c7a4c4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:31:20.021718 11805 raft_consensus.cc:504] T 00000000000000000000000000000000 P 034e3933264148c3a9b8b6b110c7a4c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "034e3933264148c3a9b8b6b110c7a4c4" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 43365 } }
I20260926 16:31:20.022192 11805 leader_election.cc:302] T 00000000000000000000000000000000 P 034e3933264148c3a9b8b6b110c7a4c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 034e3933264148c3a9b8b6b110c7a4c4; no voters:
I20260926 16:31:20.022923 11805 leader_election.cc:288] T 00000000000000000000000000000000 P 034e3933264148c3a9b8b6b110c7a4c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:31:20.023093 11810 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 034e3933264148c3a9b8b6b110c7a4c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:31:20.023860 11810 raft_consensus.cc:686] T 00000000000000000000000000000000 P 034e3933264148c3a9b8b6b110c7a4c4 [term 1 LEADER]: Becoming Leader. State: Replica: 034e3933264148c3a9b8b6b110c7a4c4, State: Running, Role: LEADER
I20260926 16:31:20.024436 11810 consensus_queue.cc:237] T 00000000000000000000000000000000 P 034e3933264148c3a9b8b6b110c7a4c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "034e3933264148c3a9b8b6b110c7a4c4" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 43365 } }
I20260926 16:31:20.025090 11805 sys_catalog.cc:564] T 00000000000000000000000000000000 P 034e3933264148c3a9b8b6b110c7a4c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:31:20.027494 11811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 034e3933264148c3a9b8b6b110c7a4c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "034e3933264148c3a9b8b6b110c7a4c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "034e3933264148c3a9b8b6b110c7a4c4" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 43365 } } }
I20260926 16:31:20.027470 11812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 034e3933264148c3a9b8b6b110c7a4c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 034e3933264148c3a9b8b6b110c7a4c4. Latest consensus state: current_term: 1 leader_uuid: "034e3933264148c3a9b8b6b110c7a4c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "034e3933264148c3a9b8b6b110c7a4c4" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 43365 } } }
I20260926 16:31:20.028023 11811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 034e3933264148c3a9b8b6b110c7a4c4 [sys.catalog]: This master's current role is: LEADER
I20260926 16:31:20.028039 11812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 034e3933264148c3a9b8b6b110c7a4c4 [sys.catalog]: This master's current role is: LEADER
I20260926 16:31:20.031777 11818 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:31:20.037400 11818 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:31:20.045962 11818 catalog_manager.cc:1347] Generated new cluster ID: 9545c0cde98c46feb8e2835af44a37db
I20260926 16:31:20.046089 11818 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:31:20.064877 11818 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:31:20.066772 11818 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:31:20.074873 11818 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 034e3933264148c3a9b8b6b110c7a4c4: Generated new TSK 0
I20260926 16:31:20.075915 11818 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:31:20.179020 11808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:20.099569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57990 (local address 127.6.33.62:43365)
0926 16:31:20.099770 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:20.099777 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:20.100417 (+ 640us) server_negotiation.cc:407] Connection header received
0926 16:31:20.100626 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:20.100632 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:20.100732 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:20.100931 (+ 199us) server_negotiation.cc:226] Negotiated authn=JWT
0926 16:31:20.101359 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:20.102490 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:20.103907 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:20.104583 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:20.104710 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:20.105220 (+ 510us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 16:31:20.178246 (+ 73026us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 16:31:20.178695 (+ 449us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57990: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":56}
W20260926 16:31:20.179145 11834 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:31:20.099558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:43365 (local address 127.0.0.1:57990)
0926 16:31:20.100313 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:20.100329 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:20.100456 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:20.100964 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:20.100968 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:20.100978 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0926 16:31:20.101221 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:20.101229 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:20.102655 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:20.102659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:20.103754 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:20.103765 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:20.104693 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:20.104743 (+ 50us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:20.104935 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:20.104969 (+ 34us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 16:31:20.178560 (+ 73591us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 16:31:20.178810 (+ 250us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.33.62:43365: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":493,"threads_started":1}
W20260926 16:31:20.179163 11808 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57990: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
I20260926 16:31:20.180891 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 11739
I20260926 16:31:20.181963 11800 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:31:20.281916 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 11739
2026-09-26T16:31:20Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (780 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (412 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (413 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:31:21 dist-test-slave-khd3 krb5kdc[11849](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:31:21 dist-test-slave-khd3 krb5kdc[11849](info): set up 2 sockets
Sep 26 16:31:21 dist-test-slave-khd3 krb5kdc[11849](info): commencing operation
krb5kdc: starting...
W20260926 16:31:22.210450 6276 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Sep 26 16:31:22 dist-test-slave-khd3 krb5kdc[11849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:31:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:31:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab.
Entry for principal kudu/127.6.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab.
I20260926 16:31:22.417104 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:46129
--webserver_interface=127.6.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:42977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.33.62:46129
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:31:22.737578 11862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:31:22.737914 11862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:31:22.738067 11862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:31:22.745556 11862 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:31:22.745710 11862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:31:22.745805 11862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:31:22.745903 11862 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:31:22.745975 11862 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:31:22.755945 11862 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:42977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.33.62:46129
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:46129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:31:22.757339 11862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:31:22.758708 11862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:31:22.770730 11867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:22.770638 11868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:22.770674 11870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:31:22.771478 11862 server_base.cc:1034] running on GCE node
I20260926 16:31:22.772293 11862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:31:22.773806 11862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:31:22.775060 11862 hybrid_clock.cc:648] HybridClock initialized: now 1790440282775013 us; error 46 us; skew 500 ppm
I20260926 16:31:22.775424 11862 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:31:22 dist-test-slave-khd3 krb5kdc[11849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440282, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:31:22.784221 11862 init.cc:377] Logged in from keytab as kudu/127.6.33.62@KRBTEST.COM (short username kudu)
I20260926 16:31:22.785821 11862 webserver.cc:460] Webserver started at http://127.6.33.62:44097/ using document root <none> and password file <none>
I20260926 16:31:22.786520 11862 fs_manager.cc:359] Metadata directory not provided
I20260926 16:31:22.786625 11862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:31:22.786880 11862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:31:22.790025 11862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/minicluster-data/master-0/data/instance:
uuid: "5d543d615e724fdcbe6f911640f4319d"
format_stamp: "Formatted at 2026-09-26 16:31:22 on dist-test-slave-khd3"
server_key: "03a8a08a344a001e8ae53fa2ff3c37b4"
server_key_iv: "5eef2599c828f1e65ee4cffe788893b6"
server_key_version: "encryptionkey@0"
I20260926 16:31:22.790778 11862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/minicluster-data/master-0/wal/instance:
uuid: "5d543d615e724fdcbe6f911640f4319d"
format_stamp: "Formatted at 2026-09-26 16:31:22 on dist-test-slave-khd3"
server_key: "03a8a08a344a001e8ae53fa2ff3c37b4"
server_key_iv: "5eef2599c828f1e65ee4cffe788893b6"
server_key_version: "encryptionkey@0"
I20260926 16:31:22.796679 11862 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260926 16:31:22.800324 11877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:22.801906 11862 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:31:22.802078 11862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/minicluster-data/master-0/wal
uuid: "5d543d615e724fdcbe6f911640f4319d"
format_stamp: "Formatted at 2026-09-26 16:31:22 on dist-test-slave-khd3"
server_key: "03a8a08a344a001e8ae53fa2ff3c37b4"
server_key_iv: "5eef2599c828f1e65ee4cffe788893b6"
server_key_version: "encryptionkey@0"
I20260926 16:31:22.802284 11862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:31:22.818915 11862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:31:22.823503 11862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:31:22.823805 11862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:31:22.842496 11862 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.62:46129
I20260926 16:31:22.842535 11928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.62:46129 every 8 connection(s)
I20260926 16:31:22.844062 11862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
I20260926 16:31:22.846777 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 11862
I20260926 16:31:22.847008 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/minicluster-data/master-0/wal/instance
I20260926 16:31:22.847769 6276 external_mini_cluster.cc:1428] Setting key 29828aa01e602a34a0cf1588d5161d9e
I20260926 16:31:22.848711 11929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:31:22.861423 11929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d543d615e724fdcbe6f911640f4319d: Bootstrap starting.
Sep 26 16:31:22 dist-test-slave-khd3 krb5kdc[11849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:31:22.865646 11929 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d543d615e724fdcbe6f911640f4319d: Neither blocks nor log segments found. Creating new log.
I20260926 16:31:22.867053 11929 log.cc:824] T 00000000000000000000000000000000 P 5d543d615e724fdcbe6f911640f4319d: Log is configured to *not* fsync() on all Append() calls
I20260926 16:31:22.870636 11929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d543d615e724fdcbe6f911640f4319d: No bootstrap required, opened a new log
I20260926 16:31:22.871845 11932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:22.849973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55308 (local address 127.6.33.62:46129)
0926 16:31:22.850758 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:22.850771 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:22.850812 (+ 41us) server_negotiation.cc:407] Connection header received
0926 16:31:22.852422 (+ 1610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:22.852449 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:22.852965 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:22.853398 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:22.854859 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:22.856233 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:22.857567 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:22.858310 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:22.858565 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:22.862794 (+ 4229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:22.862807 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:22.862827 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:22.862886 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:22.867034 (+ 4148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:22.868049 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:22.868055 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:22.868062 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:22.868236 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:22.868645 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:22.868649 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:22.868652 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:22.869375 (+ 723us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:22.869678 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:22.870059 (+ 381us) server_negotiation.cc:299] Negotiation successful
0926 16:31:22.870477 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":336,"threads_started":1}
I20260926 16:31:22.879125 11929 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d543d615e724fdcbe6f911640f4319d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d543d615e724fdcbe6f911640f4319d" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 46129 } }
I20260926 16:31:22.879469 11929 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d543d615e724fdcbe6f911640f4319d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:31:22.879555 11929 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d543d615e724fdcbe6f911640f4319d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d543d615e724fdcbe6f911640f4319d, State: Initialized, Role: FOLLOWER
I20260926 16:31:22.880096 11929 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d543d615e724fdcbe6f911640f4319d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d543d615e724fdcbe6f911640f4319d" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 46129 } }
I20260926 16:31:22.880294 11929 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d543d615e724fdcbe6f911640f4319d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:31:22.880400 11929 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d543d615e724fdcbe6f911640f4319d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:31:22.880586 11929 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d543d615e724fdcbe6f911640f4319d [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:31:22.882803 11929 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d543d615e724fdcbe6f911640f4319d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d543d615e724fdcbe6f911640f4319d" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 46129 } }
I20260926 16:31:22.883298 11929 leader_election.cc:302] T 00000000000000000000000000000000 P 5d543d615e724fdcbe6f911640f4319d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d543d615e724fdcbe6f911640f4319d; no voters:
I20260926 16:31:22.883875 11929 leader_election.cc:288] T 00000000000000000000000000000000 P 5d543d615e724fdcbe6f911640f4319d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:31:22.884030 11934 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d543d615e724fdcbe6f911640f4319d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:31:22.884701 11934 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d543d615e724fdcbe6f911640f4319d [term 1 LEADER]: Becoming Leader. State: Replica: 5d543d615e724fdcbe6f911640f4319d, State: Running, Role: LEADER
I20260926 16:31:22.885277 11934 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d543d615e724fdcbe6f911640f4319d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d543d615e724fdcbe6f911640f4319d" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 46129 } }
I20260926 16:31:22.885990 11929 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d543d615e724fdcbe6f911640f4319d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:31:22.888479 11935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d543d615e724fdcbe6f911640f4319d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d543d615e724fdcbe6f911640f4319d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d543d615e724fdcbe6f911640f4319d" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 46129 } } }
I20260926 16:31:22.888547 11936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d543d615e724fdcbe6f911640f4319d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d543d615e724fdcbe6f911640f4319d. Latest consensus state: current_term: 1 leader_uuid: "5d543d615e724fdcbe6f911640f4319d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d543d615e724fdcbe6f911640f4319d" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 46129 } } }
I20260926 16:31:22.889181 11935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d543d615e724fdcbe6f911640f4319d [sys.catalog]: This master's current role is: LEADER
I20260926 16:31:22.889228 11936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d543d615e724fdcbe6f911640f4319d [sys.catalog]: This master's current role is: LEADER
I20260926 16:31:22.890924 11943 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:31:22.896174 11943 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:31:22.905388 11943 catalog_manager.cc:1347] Generated new cluster ID: 10b4230573d54c7c8b01e8770d0826e6
I20260926 16:31:22.905560 11943 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:31:22.925702 11943 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:31:22.927294 11943 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:31:22.932788 11943 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d543d615e724fdcbe6f911640f4319d: Generated new TSK 0
I20260926 16:31:22.933861 11943 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.33.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab.
Entry for principal kudu/127.6.33.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab.
I20260926 16:31:22.985476 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.1:0
--local_ip_for_outbound_sockets=127.6.33.1
--webserver_interface=127.6.33.1
--webserver_port=0
--tserver_master_addrs=127.6.33.62:46129
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab
--principal=kudu/127.6.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:42977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:31:23.317344 11955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:31:23.317627 11955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:31:23.317719 11955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:31:23.324802 11955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:31:23.324916 11955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:31:23.325028 11955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.1
I20260926 16:31:23.335331 11955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:42977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab
--principal=kudu/127.6.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.33.1
--webserver_port=0
--tserver_master_addrs=127.6.33.62:46129
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.1
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:31:23.336562 11955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:31:23.337845 11955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:31:23.350490 11960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:23.350601 11963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:23.350510 11961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:31:23.351471 11955 server_base.cc:1034] running on GCE node
I20260926 16:31:23.352032 11955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:31:23.352926 11955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:31:23.354146 11955 hybrid_clock.cc:648] HybridClock initialized: now 1790440283354077 us; error 57 us; skew 500 ppm
I20260926 16:31:23.354489 11955 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:31:23 dist-test-slave-khd3 krb5kdc[11849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440283, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:31:23.357650 11955 init.cc:377] Logged in from keytab as kudu/127.6.33.1@KRBTEST.COM (short username kudu)
I20260926 16:31:23.359066 11955 webserver.cc:460] Webserver started at http://127.6.33.1:44491/ using document root <none> and password file <none>
I20260926 16:31:23.359745 11955 fs_manager.cc:359] Metadata directory not provided
I20260926 16:31:23.359882 11955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:31:23.360183 11955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:31:23.363188 11955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/minicluster-data/ts-0/data/instance:
uuid: "d3af4f2523b2430faab25bdbcf37e97b"
format_stamp: "Formatted at 2026-09-26 16:31:23 on dist-test-slave-khd3"
server_key: "b591f0509e0fe4284c032c8d841d729e"
server_key_iv: "ad3d68181eee705d7c2959bab696b3f1"
server_key_version: "encryptionkey@0"
I20260926 16:31:23.363948 11955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/minicluster-data/ts-0/wal/instance:
uuid: "d3af4f2523b2430faab25bdbcf37e97b"
format_stamp: "Formatted at 2026-09-26 16:31:23 on dist-test-slave-khd3"
server_key: "b591f0509e0fe4284c032c8d841d729e"
server_key_iv: "ad3d68181eee705d7c2959bab696b3f1"
server_key_version: "encryptionkey@0"
I20260926 16:31:23.369372 11955 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 16:31:23.372735 11970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:23.374212 11955 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:31:23.374366 11955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/minicluster-data/ts-0/wal
uuid: "d3af4f2523b2430faab25bdbcf37e97b"
format_stamp: "Formatted at 2026-09-26 16:31:23 on dist-test-slave-khd3"
server_key: "b591f0509e0fe4284c032c8d841d729e"
server_key_iv: "ad3d68181eee705d7c2959bab696b3f1"
server_key_version: "encryptionkey@0"
I20260926 16:31:23.374562 11955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:31:23.395241 11955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:31:23.399550 11955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:31:23.399819 11955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:31:23.400712 11955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:31:23.402017 11955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:31:23.402092 11955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:23.402243 11955 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:31:23.402300 11955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:23.434722 11955 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.1:37263
I20260926 16:31:23.434780 12082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.1:37263 every 8 connection(s)
I20260926 16:31:23.436030 11955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
I20260926 16:31:23.445783 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 11955
I20260926 16:31:23.446066 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/minicluster-data/ts-0/wal/instance
I20260926 16:31:23.446796 6276 external_mini_cluster.cc:1428] Setting key 9fbbda7ab425ce02662906a7ae3758b4
Sep 26 16:31:23 dist-test-slave-khd3 krb5kdc[11849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440283, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.1@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:31:23.458627 12085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:23.438652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:46031 (local address 127.6.33.62:46129)
0926 16:31:23.439102 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:23.439106 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:23.440092 (+ 986us) server_negotiation.cc:407] Connection header received
0926 16:31:23.442086 (+ 1994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:23.442092 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:23.442187 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:23.442300 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:23.443863 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:23.444672 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:23.446965 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:23.447432 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:23.447575 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:23.451211 (+ 3636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:23.451220 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:23.451227 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:23.451275 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:23.454960 (+ 3685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:23.456119 (+ 1159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:23.456124 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:23.456126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:23.456256 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:23.456830 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:23.456834 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:23.456836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:23.457105 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:23.457216 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:23.458140 (+ 924us) server_negotiation.cc:299] Negotiation successful
0926 16:31:23.458314 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":252,"threads_started":1}
I20260926 16:31:23.460254 12086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:23.439206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:46129 (local address 127.6.33.1:46031)
0926 16:31:23.439932 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:23.439972 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:23.441771 (+ 1799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:23.442543 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:23.442552 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:23.443118 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:23.443697 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:23.443714 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:23.444858 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:23.444861 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:23.446759 (+ 1898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:23.446773 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:23.447660 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:23.447665 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:23.447903 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:23.448744 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:23.448814 (+ 70us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:23.450961 (+ 2147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:31:23.455180 (+ 4219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:23.455191 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:23.455218 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:23.455952 (+ 734us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:23.456387 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:23.456392 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:23.456396 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:23.456670 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:23.457228 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:31:23.457240 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:31:23.457734 (+ 494us) client_negotiation.cc:769] Sending connection context
0926 16:31:23.458062 (+ 328us) client_negotiation.cc:240] Negotiation successful
0926 16:31:23.458689 (+ 627us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":302,"threads_started":1}
I20260926 16:31:23.461920 12083 heartbeater.cc:347] Connected to a master server at 127.6.33.62:46129
I20260926 16:31:23.462241 12083 heartbeater.cc:464] Registering TS with master...
I20260926 16:31:23.462975 12083 heartbeater.cc:511] Master 127.6.33.62:46129 requested a full tablet report, sending...
I20260926 16:31:23.465178 11894 ts_manager.cc:194] Registered new tserver with Master: d3af4f2523b2430faab25bdbcf37e97b (127.6.33.1:37263)
I20260926 16:31:23.466593 11894 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.33.1@KRBTEST.COM'} at 127.6.33.1:46031
WARNING: no policy specified for kudu/127.6.33.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab.
Entry for principal kudu/127.6.33.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab.
I20260926 16:31:23.493896 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.2:0
--local_ip_for_outbound_sockets=127.6.33.2
--webserver_interface=127.6.33.2
--webserver_port=0
--tserver_master_addrs=127.6.33.62:46129
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab
--principal=kudu/127.6.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:42977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:31:23.822438 12089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:31:23.822764 12089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:31:23.822928 12089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:31:23.830241 12089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:31:23.830417 12089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:31:23.830597 12089 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.2
I20260926 16:31:23.840351 12089 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:42977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab
--principal=kudu/127.6.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.33.2
--webserver_port=0
--tserver_master_addrs=127.6.33.62:46129
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.2
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:31:23.841887 12089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:31:23.843210 12089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:31:23.856216 12094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:23.856190 12095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:23.857182 12097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:31:23.857940 12089 server_base.cc:1034] running on GCE node
I20260926 16:31:23.858646 12089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:31:23.859577 12089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:31:23.860838 12089 hybrid_clock.cc:648] HybridClock initialized: now 1790440283860746 us; error 26 us; skew 500 ppm
I20260926 16:31:23.861155 12089 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:31:23 dist-test-slave-khd3 krb5kdc[11849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440283, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:31:23.864398 12089 init.cc:377] Logged in from keytab as kudu/127.6.33.2@KRBTEST.COM (short username kudu)
I20260926 16:31:23.865972 12089 webserver.cc:460] Webserver started at http://127.6.33.2:39325/ using document root <none> and password file <none>
I20260926 16:31:23.866662 12089 fs_manager.cc:359] Metadata directory not provided
I20260926 16:31:23.866766 12089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:31:23.867033 12089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:31:23.870048 12089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/minicluster-data/ts-1/data/instance:
uuid: "6205548d04bf49e28fc479ade132a69e"
format_stamp: "Formatted at 2026-09-26 16:31:23 on dist-test-slave-khd3"
server_key: "c15294da3cf54aae5861ef2b180b00ec"
server_key_iv: "5e3d7545a6b69e644668a015eb39a8fe"
server_key_version: "encryptionkey@0"
I20260926 16:31:23.870800 12089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/minicluster-data/ts-1/wal/instance:
uuid: "6205548d04bf49e28fc479ade132a69e"
format_stamp: "Formatted at 2026-09-26 16:31:23 on dist-test-slave-khd3"
server_key: "c15294da3cf54aae5861ef2b180b00ec"
server_key_iv: "5e3d7545a6b69e644668a015eb39a8fe"
server_key_version: "encryptionkey@0"
I20260926 16:31:23.876052 12089 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 16:31:23.879830 12104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:23.881330 12089 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:31:23.881568 12089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/minicluster-data/ts-1/wal
uuid: "6205548d04bf49e28fc479ade132a69e"
format_stamp: "Formatted at 2026-09-26 16:31:23 on dist-test-slave-khd3"
server_key: "c15294da3cf54aae5861ef2b180b00ec"
server_key_iv: "5e3d7545a6b69e644668a015eb39a8fe"
server_key_version: "encryptionkey@0"
I20260926 16:31:23.881800 12089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:31:23.910099 12089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:31:23.914393 12089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:31:23.914808 12089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:31:23.915796 12089 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:31:23.917189 12089 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:31:23.917260 12089 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:23.917402 12089 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:31:23.917450 12089 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:23.949302 12089 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.2:40461
I20260926 16:31:23.949370 12216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.2:40461 every 8 connection(s)
I20260926 16:31:23.950525 12089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
I20260926 16:31:23.953462 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 12089
I20260926 16:31:23.953648 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/minicluster-data/ts-1/wal/instance
I20260926 16:31:23.954093 6276 external_mini_cluster.cc:1428] Setting key eb78bef016df6084724bc50132212ac6
Sep 26 16:31:23 dist-test-slave-khd3 krb5kdc[11849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440283, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.2@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:31:23.973054 12085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:23.953172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:33013 (local address 127.6.33.62:46129)
0926 16:31:23.953375 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:23.953381 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:23.954795 (+ 1414us) server_negotiation.cc:407] Connection header received
0926 16:31:23.957098 (+ 2303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:23.957102 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:23.957166 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:23.957254 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:23.959390 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:23.960176 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:23.962753 (+ 2577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:23.963202 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:23.963362 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:23.967208 (+ 3846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:23.967214 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:23.967217 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:23.967250 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:23.969973 (+ 2723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:23.970745 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:23.970750 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:23.970752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:23.970875 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:23.971304 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:23.971314 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:23.971316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:23.971590 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:23.971745 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:23.972455 (+ 710us) server_negotiation.cc:299] Negotiation successful
0926 16:31:23.972678 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 16:31:23.974079 12219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:23.953778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:46129 (local address 127.6.33.2:33013)
0926 16:31:23.954611 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:23.954670 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:23.956746 (+ 2076us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:23.957543 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:23.957558 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:23.958371 (+ 813us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:23.959195 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:23.959222 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:23.960325 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:23.960333 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:23.962575 (+ 2242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:23.962597 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:23.963448 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:23.963457 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:23.963716 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:23.964448 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:23.964473 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:23.966966 (+ 2493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:31:23.970138 (+ 3172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:23.970149 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:23.970168 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:23.970600 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:23.970992 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:23.970997 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:23.971000 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:23.971187 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:23.971713 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:31:23.971722 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:31:23.972078 (+ 356us) client_negotiation.cc:769] Sending connection context
0926 16:31:23.972344 (+ 266us) client_negotiation.cc:240] Negotiation successful
0926 16:31:23.972930 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":317,"threads_started":1}
WARNING: no policy specified for kudu/127.6.33.3@KRBTEST.COM; defaulting to no policy
I20260926 16:31:23.975489 12217 heartbeater.cc:347] Connected to a master server at 127.6.33.62:46129
I20260926 16:31:23.975847 12217 heartbeater.cc:464] Registering TS with master...
I20260926 16:31:23.976624 12217 heartbeater.cc:511] Master 127.6.33.62:46129 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.3@KRBTEST.COM" created.
I20260926 16:31:23.978204 11894 ts_manager.cc:194] Registered new tserver with Master: 6205548d04bf49e28fc479ade132a69e (127.6.33.2:40461)
I20260926 16:31:23.979005 11894 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.33.2@KRBTEST.COM'} at 127.6.33.2:33013
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab.
Entry for principal kudu/127.6.33.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab.
I20260926 16:31:24.004117 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.3:0
--local_ip_for_outbound_sockets=127.6.33.3
--webserver_interface=127.6.33.3
--webserver_port=0
--tserver_master_addrs=127.6.33.62:46129
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab
--principal=kudu/127.6.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:42977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:31:24.329457 12222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:31:24.329737 12222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:31:24.329911 12222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:31:24.337244 12222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:31:24.337371 12222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:31:24.337559 12222 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.3
I20260926 16:31:24.347852 12222 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:42977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab
--principal=kudu/127.6.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.33.3
--webserver_port=0
--tserver_master_addrs=127.6.33.62:46129
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.3
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:31:24.349208 12222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:31:24.350466 12222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:31:24.363291 12227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:24.363368 12228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:24.363346 12230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:31:24.364538 12222 server_base.cc:1034] running on GCE node
I20260926 16:31:24.365149 12222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:31:24.366055 12222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:31:24.367271 12222 hybrid_clock.cc:648] HybridClock initialized: now 1790440284367228 us; error 54 us; skew 500 ppm
I20260926 16:31:24.367580 12222 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:31:24 dist-test-slave-khd3 krb5kdc[11849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440284, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:31:24.370685 12222 init.cc:377] Logged in from keytab as kudu/127.6.33.3@KRBTEST.COM (short username kudu)
I20260926 16:31:24.372175 12222 webserver.cc:460] Webserver started at http://127.6.33.3:32833/ using document root <none> and password file <none>
I20260926 16:31:24.373124 12222 fs_manager.cc:359] Metadata directory not provided
I20260926 16:31:24.373284 12222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:31:24.373613 12222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:31:24.376381 12222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/minicluster-data/ts-2/data/instance:
uuid: "8dc9e8cf5a894ffaa69d8407aefd300e"
format_stamp: "Formatted at 2026-09-26 16:31:24 on dist-test-slave-khd3"
server_key: "5e2d2005ec09e253b605ec2f6fd812cf"
server_key_iv: "7da22c5f0d7917808c7891866b1c5acc"
server_key_version: "encryptionkey@0"
I20260926 16:31:24.377156 12222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/minicluster-data/ts-2/wal/instance:
uuid: "8dc9e8cf5a894ffaa69d8407aefd300e"
format_stamp: "Formatted at 2026-09-26 16:31:24 on dist-test-slave-khd3"
server_key: "5e2d2005ec09e253b605ec2f6fd812cf"
server_key_iv: "7da22c5f0d7917808c7891866b1c5acc"
server_key_version: "encryptionkey@0"
I20260926 16:31:24.382170 12222 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 16:31:24.385528 12237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:24.387195 12222 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:31:24.387352 12222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/minicluster-data/ts-2/wal
uuid: "8dc9e8cf5a894ffaa69d8407aefd300e"
format_stamp: "Formatted at 2026-09-26 16:31:24 on dist-test-slave-khd3"
server_key: "5e2d2005ec09e253b605ec2f6fd812cf"
server_key_iv: "7da22c5f0d7917808c7891866b1c5acc"
server_key_version: "encryptionkey@0"
I20260926 16:31:24.387552 12222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:31:24.411664 12222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:31:24.415676 12222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:31:24.415972 12222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:31:24.416970 12222 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:31:24.418273 12222 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:31:24.418406 12222 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:24.418550 12222 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:31:24.418617 12222 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:24.451800 12222 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.3:33267
I20260926 16:31:24.451838 12349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.3:33267 every 8 connection(s)
I20260926 16:31:24.453264 12222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
I20260926 16:31:24.454698 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 12222
I20260926 16:31:24.454890 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/minicluster-data/ts-2/wal/instance
I20260926 16:31:24.455324 6276 external_mini_cluster.cc:1428] Setting key 74070a2fc623c8799c2fc60545f238e5
Sep 26 16:31:24 dist-test-slave-khd3 krb5kdc[11849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440284, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.3@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:31:24.469671 12083 heartbeater.cc:503] Master 127.6.33.62:46129 was elected leader, sending a full tablet report...
I20260926 16:31:24.475349 12085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:24.456185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.3:58183 (local address 127.6.33.62:46129)
0926 16:31:24.456370 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:24.456376 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:24.457862 (+ 1486us) server_negotiation.cc:407] Connection header received
0926 16:31:24.459904 (+ 2042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:24.459908 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:24.459984 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:24.460068 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:24.462186 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:24.463010 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:24.464609 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:24.465139 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:24.465246 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:24.468952 (+ 3706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:24.468959 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:24.468963 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:24.469005 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:24.472281 (+ 3276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:24.473125 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:24.473129 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:24.473132 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:24.473249 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:24.473711 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:24.473714 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:24.473716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:24.473966 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:24.474096 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:24.474786 (+ 690us) server_negotiation.cc:299] Negotiation successful
0926 16:31:24.475057 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 16:31:24.476270 12352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:24.456826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:46129 (local address 127.6.33.3:58183)
0926 16:31:24.457699 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:24.457749 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:24.459608 (+ 1859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:24.460263 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:24.460272 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:24.461124 (+ 852us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:24.461977 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:24.462004 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:24.463165 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:24.463169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:24.464449 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:24.464461 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:24.465334 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:24.465338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:24.465550 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:24.466208 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:24.466232 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:24.468622 (+ 2390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:31:24.472423 (+ 3801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:24.472430 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:24.472445 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:24.473017 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:24.473344 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:24.473347 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:24.473350 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:24.473615 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:24.474106 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:31:24.474112 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:31:24.474450 (+ 338us) client_negotiation.cc:769] Sending connection context
0926 16:31:24.474684 (+ 234us) client_negotiation.cc:240] Negotiation successful
0926 16:31:24.475125 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":352,"threads_started":1}
I20260926 16:31:24.477705 12350 heartbeater.cc:347] Connected to a master server at 127.6.33.62:46129
I20260926 16:31:24.478016 12350 heartbeater.cc:464] Registering TS with master...
I20260926 16:31:24.478809 12350 heartbeater.cc:511] Master 127.6.33.62:46129 requested a full tablet report, sending...
I20260926 16:31:24.480232 11894 ts_manager.cc:194] Registered new tserver with Master: 8dc9e8cf5a894ffaa69d8407aefd300e (127.6.33.3:33267)
I20260926 16:31:24.481099 11894 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.33.3@KRBTEST.COM'} at 127.6.33.3:58183
I20260926 16:31:24.483421 6276 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/krb5kdc/test-admin.keytab.
Sep 26 16:31:24 dist-test-slave-khd3 krb5kdc[11849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:31:24.509691 6276 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260926 16:31:24.514703 12357 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:31:24.512263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46934 (local address 127.6.33.1:37263)
0926 16:31:24.512955 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:24.512962 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:24.512990 (+ 28us) server_negotiation.cc:407] Connection header received
0926 16:31:24.513050 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:24.513056 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:24.513262 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:24.513405 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:24.514265 (+ 860us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46934: BlockingRecv error: recv got EOF from 127.0.0.1:46934 (error 108)
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":367,"threads_started":1}
W20260926 16:31:24.515039 6276 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/krb5kdc/krb5cc' of type FILE
Sep 26 16:31:24 dist-test-slave-khd3 krb5kdc[11849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:31:24.516454 6276 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 16:31:24 dist-test-slave-khd3 krb5kdc[11849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:31:24.517778 6276 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 16:31:24 dist-test-slave-khd3 krb5kdc[11849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.1@KRBTEST.COM
I20260926 16:31:24.533332 12357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:24.519659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46938 (local address 127.6.33.1:37263)
0926 16:31:24.519850 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:24.519854 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:24.519990 (+ 136us) server_negotiation.cc:407] Connection header received
0926 16:31:24.520151 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:24.520155 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:24.520211 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:24.520285 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:24.521354 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:24.522351 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:24.523557 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:24.524100 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:24.524192 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:24.527834 (+ 3642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:24.527845 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:24.527860 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:24.527889 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:24.530869 (+ 2980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:24.531455 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:24.531462 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:24.531464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:24.531585 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:24.531906 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:24.531910 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:24.531912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:24.532256 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:24.532458 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:24.532731 (+ 273us) server_negotiation.cc:299] Negotiation successful
0926 16:31:24.533035 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260926 16:31:24.539217 12357 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:31:24.537450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46942 (local address 127.6.33.1:37263)
0926 16:31:24.537677 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:24.537683 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:24.537907 (+ 224us) server_negotiation.cc:407] Connection header received
0926 16:31:24.538098 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:24.538103 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:24.538159 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:24.538255 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:24.538991 (+ 736us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46942: BlockingRecv error: recv got EOF from 127.0.0.1:46942 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260926 16:31:24.539842 6276 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/krb5kdc/krb5cc)
Sep 26 16:31:24 dist-test-slave-khd3 krb5kdc[11849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:31:24.541395 6276 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 16:31:24 dist-test-slave-khd3 krb5kdc[11849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:31:24.542716 6276 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 16:31:24 dist-test-slave-khd3 krb5kdc[11849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.1@KRBTEST.COM
I20260926 16:31:24.557814 12357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:24.543962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46952 (local address 127.6.33.1:37263)
0926 16:31:24.544152 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:24.544161 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:24.544446 (+ 285us) server_negotiation.cc:407] Connection header received
0926 16:31:24.544599 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:24.544603 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:24.544660 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:24.544743 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:24.545739 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:24.546405 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:24.547637 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:24.548080 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:24.548186 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:24.552274 (+ 4088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:24.552279 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:24.552283 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:24.552329 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:24.554825 (+ 2496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:24.555602 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:24.555606 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:24.555608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:24.555731 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:24.556079 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:24.556084 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:24.556086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:24.556417 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:24.556633 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:24.557182 (+ 549us) server_negotiation.cc:299] Negotiation successful
0926 16:31:24.557488 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260926 16:31:24.562515 12357 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:31:24.560689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46960 (local address 127.6.33.1:37263)
0926 16:31:24.560864 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:24.560868 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:24.561123 (+ 255us) server_negotiation.cc:407] Connection header received
0926 16:31:24.561253 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:24.561257 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:24.561328 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:24.561415 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:24.562174 (+ 759us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46960: BlockingRecv error: recv got EOF from 127.0.0.1:46960 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260926 16:31:24.562682 6276 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440233850748-6276-0/krb5kdc/krb5cc' of type FILE
Sep 26 16:31:24 dist-test-slave-khd3 krb5kdc[11849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:31:24.564256 6276 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 16:31:24 dist-test-slave-khd3 krb5kdc[11849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:31:24.565753 6276 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 16:31:24 dist-test-slave-khd3 krb5kdc[11849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.1@KRBTEST.COM
I20260926 16:31:24.580183 12357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:24.567166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46970 (local address 127.6.33.1:37263)
0926 16:31:24.567412 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:24.567418 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:24.567678 (+ 260us) server_negotiation.cc:407] Connection header received
0926 16:31:24.567809 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:24.567813 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:24.567875 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:24.567953 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:24.569029 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:24.569733 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:24.570998 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:24.571408 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:24.571511 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:24.575073 (+ 3562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:24.575078 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:24.575081 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:24.575116 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:24.577584 (+ 2468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:24.578240 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:24.578246 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:24.578250 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:24.578439 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:24.578810 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:24.578815 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:24.578817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:24.579075 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:24.579390 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:24.579650 (+ 260us) server_negotiation.cc:299] Negotiation successful
0926 16:31:24.579855 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 16:31:24.582063 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 11955
I20260926 16:31:24.594455 12357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:24.582781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46974 (local address 127.6.33.1:37263)
0926 16:31:24.582928 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:24.582933 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:24.583322 (+ 389us) server_negotiation.cc:407] Connection header received
0926 16:31:24.583609 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:24.583613 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:24.583674 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:24.583757 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:24.584691 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:24.585551 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:24.586917 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:24.587328 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:24.587428 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:24.589508 (+ 2080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:24.589513 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:24.589516 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:24.589546 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:24.592180 (+ 2634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:24.592867 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:24.592872 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:24.592875 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:24.593011 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:24.593291 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:24.593293 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:24.593295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:24.593572 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:24.593759 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:24.593998 (+ 239us) server_negotiation.cc:299] Negotiation successful
0926 16:31:24.594158 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 16:31:24.595012 12078 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:31:24.679203 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 11955
I20260926 16:31:24.696147 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 12089
Sep 26 16:31:24 dist-test-slave-khd3 krb5kdc[11849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.2@KRBTEST.COM
I20260926 16:31:24.712576 12370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:24.697242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42610 (local address 127.6.33.2:40461)
0926 16:31:24.697818 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:24.697825 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:24.697852 (+ 27us) server_negotiation.cc:407] Connection header received
0926 16:31:24.697910 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:24.697917 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:24.698172 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:24.698301 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:24.699355 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:24.700441 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:24.701826 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:24.702400 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:24.702516 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:24.706615 (+ 4099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:24.706629 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:24.706644 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:24.706684 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:24.709798 (+ 3114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:24.710533 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:24.710540 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:24.710542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:24.710664 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:24.711060 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:24.711064 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:24.711066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:24.711425 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:24.711667 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:24.711962 (+ 295us) server_negotiation.cc:299] Negotiation successful
0926 16:31:24.712273 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":321,"threads_started":1}
I20260926 16:31:24.714010 12212 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:31:24.795881 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 12089
I20260926 16:31:24.811667 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 12222
Sep 26 16:31:24 dist-test-slave-khd3 krb5kdc[11849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.3@KRBTEST.COM
I20260926 16:31:24.827090 12372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:24.812794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34876 (local address 127.6.33.3:33267)
0926 16:31:24.813457 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:24.813464 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:24.813497 (+ 33us) server_negotiation.cc:407] Connection header received
0926 16:31:24.813559 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:24.813564 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:24.813773 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:24.813900 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:24.814926 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:24.815952 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:24.817354 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:24.817899 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:24.818002 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:24.821940 (+ 3938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:24.821950 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:24.821963 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:24.822029 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:24.824800 (+ 2771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:24.825412 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:24.825419 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:24.825421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:24.825537 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:24.825868 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:24.825871 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:24.825873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:24.826178 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:24.826401 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:24.826613 (+ 212us) server_negotiation.cc:299] Negotiation successful
0926 16:31:24.826795 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":369,"threads_started":1}
I20260926 16:31:24.828394 12345 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:31:24.915596 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 12222
I20260926 16:31:24.930647 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 11862
I20260926 16:31:24.931751 11924 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:31:25.034294 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 11862
2026-09-26T16:31:25Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3933 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:31:25 dist-test-slave-khd3 krb5kdc[12377](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:31:25 dist-test-slave-khd3 krb5kdc[12377](info): set up 2 sockets
Sep 26 16:31:25 dist-test-slave-khd3 krb5kdc[12377](info): commencing operation
krb5kdc: starting...
W20260926 16:31:26.126456 6276 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.010s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Sep 26 16:31:26 dist-test-slave-khd3 krb5kdc[12377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:31:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:31:26Z Disabled control of system clock
WARNING: no policy specified for oryx/127.6.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/krb5kdc/oryx_127.6.33.62.keytab.
Entry for principal oryx/127.6.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/krb5kdc/oryx_127.6.33.62.keytab.
I20260926 16:31:26.342736 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:33939
--webserver_interface=127.6.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/krb5kdc/oryx_127.6.33.62.keytab
--principal=oryx/127.6.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:42049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.33.62:33939
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:31:26.667271 12390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:31:26.667584 12390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:31:26.667761 12390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:31:26.675177 12390 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:31:26.675287 12390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:31:26.675380 12390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:31:26.675465 12390 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:31:26.675557 12390 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:31:26.685352 12390 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:42049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.33.62:33939
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/krb5kdc/oryx_127.6.33.62.keytab
--principal=oryx/127.6.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:33939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:31:26.686728 12390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:31:26.688072 12390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:31:26.700536 12398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:26.700570 12396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:26.700500 12395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:31:26.701117 12390 server_base.cc:1034] running on GCE node
I20260926 16:31:26.702072 12390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:31:26.703672 12390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:31:26.704907 12390 hybrid_clock.cc:648] HybridClock initialized: now 1790440286704846 us; error 51 us; skew 500 ppm
I20260926 16:31:26.705286 12390 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:31:26 dist-test-slave-khd3 krb5kdc[12377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440286, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:31:26.708457 12390 init.cc:377] Logged in from keytab as oryx/127.6.33.62@KRBTEST.COM (short username oryx)
I20260926 16:31:26.710201 12390 webserver.cc:460] Webserver started at http://127.6.33.62:43809/ using document root <none> and password file <none>
I20260926 16:31:26.710943 12390 fs_manager.cc:359] Metadata directory not provided
I20260926 16:31:26.711091 12390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:31:26.711395 12390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:31:26.714586 12390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/master-0/data/instance:
uuid: "7a7973add6e4409dabfab1a4d9e6b175"
format_stamp: "Formatted at 2026-09-26 16:31:26 on dist-test-slave-khd3"
server_key: "947400d1a4c7b42a4161c1ed136bb673"
server_key_iv: "fb323a8043f8265196d6c7d9101b0fbb"
server_key_version: "encryptionkey@0"
I20260926 16:31:26.715426 12390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/master-0/wal/instance:
uuid: "7a7973add6e4409dabfab1a4d9e6b175"
format_stamp: "Formatted at 2026-09-26 16:31:26 on dist-test-slave-khd3"
server_key: "947400d1a4c7b42a4161c1ed136bb673"
server_key_iv: "fb323a8043f8265196d6c7d9101b0fbb"
server_key_version: "encryptionkey@0"
I20260926 16:31:26.720679 12390 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 16:31:26.724298 12405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:26.725735 12390 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 16:31:26.725922 12390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/master-0/wal
uuid: "7a7973add6e4409dabfab1a4d9e6b175"
format_stamp: "Formatted at 2026-09-26 16:31:26 on dist-test-slave-khd3"
server_key: "947400d1a4c7b42a4161c1ed136bb673"
server_key_iv: "fb323a8043f8265196d6c7d9101b0fbb"
server_key_version: "encryptionkey@0"
I20260926 16:31:26.726116 12390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:31:26.747320 12390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:31:26.751616 12390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:31:26.751905 12390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:31:26.770251 12390 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.62:33939
I20260926 16:31:26.770282 12456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.62:33939 every 8 connection(s)
I20260926 16:31:26.771627 12390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
I20260926 16:31:26.773257 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 12390
I20260926 16:31:26.773485 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/master-0/wal/instance
I20260926 16:31:26.774340 6276 external_mini_cluster.cc:1428] Setting key be5e2afb8eed9e006b4bebc739419c59
I20260926 16:31:26.776546 12457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 16:31:26 dist-test-slave-khd3 krb5kdc[12377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM
I20260926 16:31:26.790747 12457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a7973add6e4409dabfab1a4d9e6b175: Bootstrap starting.
I20260926 16:31:26.794368 12457 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a7973add6e4409dabfab1a4d9e6b175: Neither blocks nor log segments found. Creating new log.
I20260926 16:31:26.795683 12457 log.cc:824] T 00000000000000000000000000000000 P 7a7973add6e4409dabfab1a4d9e6b175: Log is configured to *not* fsync() on all Append() calls
I20260926 16:31:26.799084 12460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:26.776402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50068 (local address 127.6.33.62:33939)
0926 16:31:26.777266 (+ 864us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:26.777277 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:26.777317 (+ 40us) server_negotiation.cc:407] Connection header received
0926 16:31:26.778878 (+ 1561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:26.778898 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:26.779286 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:26.779629 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:26.781509 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:26.782905 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:26.784654 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:26.785516 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:26.785657 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:26.790636 (+ 4979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:26.790645 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:26.790660 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:26.790704 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:26.794209 (+ 3505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:26.795133 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:26.795140 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:26.795147 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:26.795325 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:26.795836 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:26.795843 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:26.795848 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:26.796821 (+ 973us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:26.797182 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:26.797543 (+ 361us) server_negotiation.cc:299] Negotiation successful
0926 16:31:26.797977 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":671,"thread_start_us":431,"threads_started":1}
I20260926 16:31:26.799141 12457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a7973add6e4409dabfab1a4d9e6b175: No bootstrap required, opened a new log
I20260926 16:31:26.806507 12457 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a7973add6e4409dabfab1a4d9e6b175 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a7973add6e4409dabfab1a4d9e6b175" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 33939 } }
I20260926 16:31:26.806851 12457 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a7973add6e4409dabfab1a4d9e6b175 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:31:26.806977 12457 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a7973add6e4409dabfab1a4d9e6b175 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a7973add6e4409dabfab1a4d9e6b175, State: Initialized, Role: FOLLOWER
I20260926 16:31:26.807554 12457 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a7973add6e4409dabfab1a4d9e6b175 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a7973add6e4409dabfab1a4d9e6b175" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 33939 } }
I20260926 16:31:26.807838 12457 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a7973add6e4409dabfab1a4d9e6b175 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:31:26.808000 12457 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a7973add6e4409dabfab1a4d9e6b175 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:31:26.808203 12457 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a7973add6e4409dabfab1a4d9e6b175 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:31:26.810555 12457 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a7973add6e4409dabfab1a4d9e6b175 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a7973add6e4409dabfab1a4d9e6b175" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 33939 } }
I20260926 16:31:26.811079 12457 leader_election.cc:302] T 00000000000000000000000000000000 P 7a7973add6e4409dabfab1a4d9e6b175 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a7973add6e4409dabfab1a4d9e6b175; no voters:
I20260926 16:31:26.811646 12457 leader_election.cc:288] T 00000000000000000000000000000000 P 7a7973add6e4409dabfab1a4d9e6b175 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:31:26.811786 12462 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a7973add6e4409dabfab1a4d9e6b175 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:31:26.813112 12462 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a7973add6e4409dabfab1a4d9e6b175 [term 1 LEADER]: Becoming Leader. State: Replica: 7a7973add6e4409dabfab1a4d9e6b175, State: Running, Role: LEADER
I20260926 16:31:26.813632 12462 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a7973add6e4409dabfab1a4d9e6b175 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a7973add6e4409dabfab1a4d9e6b175" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 33939 } }
I20260926 16:31:26.813891 12457 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a7973add6e4409dabfab1a4d9e6b175 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:31:26.816717 12463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a7973add6e4409dabfab1a4d9e6b175 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a7973add6e4409dabfab1a4d9e6b175" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a7973add6e4409dabfab1a4d9e6b175" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 33939 } } }
I20260926 16:31:26.817101 12463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a7973add6e4409dabfab1a4d9e6b175 [sys.catalog]: This master's current role is: LEADER
I20260926 16:31:26.816712 12464 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a7973add6e4409dabfab1a4d9e6b175 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a7973add6e4409dabfab1a4d9e6b175. Latest consensus state: current_term: 1 leader_uuid: "7a7973add6e4409dabfab1a4d9e6b175" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a7973add6e4409dabfab1a4d9e6b175" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 33939 } } }
I20260926 16:31:26.817922 12464 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a7973add6e4409dabfab1a4d9e6b175 [sys.catalog]: This master's current role is: LEADER
I20260926 16:31:26.819095 12469 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:31:26.824461 12469 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:31:26.833655 12469 catalog_manager.cc:1347] Generated new cluster ID: cb13d77c4ee3430abf37211ff83e1ee5
I20260926 16:31:26.833767 12469 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:31:26.840938 12469 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:31:26.842793 12469 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:31:26.850262 12469 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a7973add6e4409dabfab1a4d9e6b175: Generated new TSK 0
I20260926 16:31:26.851351 12469 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.6.33.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.33.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.33.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/krb5kdc/oryx_127.6.33.1.keytab.
Entry for principal oryx/127.6.33.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/krb5kdc/oryx_127.6.33.1.keytab.
I20260926 16:31:26.913987 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.1:0
--local_ip_for_outbound_sockets=127.6.33.1
--webserver_interface=127.6.33.1
--webserver_port=0
--tserver_master_addrs=127.6.33.62:33939
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/krb5kdc/oryx_127.6.33.1.keytab
--principal=oryx/127.6.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:42049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:31:27.250947 12483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:31:27.251271 12483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:31:27.251447 12483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:31:27.258826 12483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:31:27.259003 12483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:31:27.259189 12483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.1
I20260926 16:31:27.269670 12483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:42049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/krb5kdc/oryx_127.6.33.1.keytab
--principal=oryx/127.6.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.33.1
--webserver_port=0
--tserver_master_addrs=127.6.33.62:33939
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.1
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:31:27.271072 12483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:31:27.272457 12483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:31:27.285895 12488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:27.285873 12489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:27.285948 12491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:31:27.287169 12483 server_base.cc:1034] running on GCE node
I20260926 16:31:27.287801 12483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:31:27.288709 12483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:31:27.289954 12483 hybrid_clock.cc:648] HybridClock initialized: now 1790440287289909 us; error 38 us; skew 500 ppm
I20260926 16:31:27.290285 12483 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:31:27 dist-test-slave-khd3 krb5kdc[12377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440287, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:31:27.293547 12483 init.cc:377] Logged in from keytab as oryx/127.6.33.1@KRBTEST.COM (short username oryx)
I20260926 16:31:27.295195 12483 webserver.cc:460] Webserver started at http://127.6.33.1:36175/ using document root <none> and password file <none>
I20260926 16:31:27.295986 12483 fs_manager.cc:359] Metadata directory not provided
I20260926 16:31:27.296128 12483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:31:27.296450 12483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:31:27.299571 12483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-0/data/instance:
uuid: "b1109a56871a442584d8a397a4c72542"
format_stamp: "Formatted at 2026-09-26 16:31:27 on dist-test-slave-khd3"
server_key: "6f8774f1fab95e3d5bf05d142b51e3cd"
server_key_iv: "577613021127d607389ee5aeb2fe8fc1"
server_key_version: "encryptionkey@0"
I20260926 16:31:27.300323 12483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-0/wal/instance:
uuid: "b1109a56871a442584d8a397a4c72542"
format_stamp: "Formatted at 2026-09-26 16:31:27 on dist-test-slave-khd3"
server_key: "6f8774f1fab95e3d5bf05d142b51e3cd"
server_key_iv: "577613021127d607389ee5aeb2fe8fc1"
server_key_version: "encryptionkey@0"
I20260926 16:31:27.305809 12483 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 16:31:27.309341 12498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:27.310773 12483 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:31:27.310969 12483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-0/wal
uuid: "b1109a56871a442584d8a397a4c72542"
format_stamp: "Formatted at 2026-09-26 16:31:27 on dist-test-slave-khd3"
server_key: "6f8774f1fab95e3d5bf05d142b51e3cd"
server_key_iv: "577613021127d607389ee5aeb2fe8fc1"
server_key_version: "encryptionkey@0"
I20260926 16:31:27.311178 12483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:31:27.350749 12483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:31:27.355019 12483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:31:27.355319 12483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:31:27.356564 12483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:31:27.357980 12483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:31:27.358054 12483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:27.358135 12483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:31:27.358178 12483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:27.391424 12483 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.1:38529
I20260926 16:31:27.391521 12610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.1:38529 every 8 connection(s)
I20260926 16:31:27.392726 12483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
I20260926 16:31:27.397612 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 12483
I20260926 16:31:27.397842 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-0/wal/instance
I20260926 16:31:27.398451 6276 external_mini_cluster.cc:1428] Setting key 45ad5edbd093741771da773e017bc9e7
Sep 26 16:31:27 dist-test-slave-khd3 krb5kdc[12377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440287, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.33.1@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM
I20260926 16:31:27.413355 12613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:27.395351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:39519 (local address 127.6.33.62:33939)
0926 16:31:27.395799 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:27.395804 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:27.396662 (+ 858us) server_negotiation.cc:407] Connection header received
0926 16:31:27.398761 (+ 2099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:27.398768 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:27.398862 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:27.398980 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:27.400825 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:27.401524 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:27.403071 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:27.403446 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:27.403577 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:27.407362 (+ 3785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:27.407368 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:27.407371 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:27.407403 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:27.410187 (+ 2784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:27.410958 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:27.410964 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:27.410967 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:27.411113 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:27.411549 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:27.411555 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:27.411558 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:27.411840 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:27.412001 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:27.412716 (+ 715us) server_negotiation.cc:299] Negotiation successful
0926 16:31:27.412983 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":245,"threads_started":1}
I20260926 16:31:27.414530 12614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:27.395791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:33939 (local address 127.6.33.1:39519)
0926 16:31:27.396501 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:27.396538 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:27.398445 (+ 1907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:27.399227 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:27.399235 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:27.399981 (+ 746us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:27.400636 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:27.400653 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:27.401685 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:27.401691 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:27.402916 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:27.402928 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:27.403678 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:27.403686 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:27.403918 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:27.404744 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:27.404853 (+ 109us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:27.407110 (+ 2257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:31:27.410344 (+ 3234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:27.410354 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:27.410373 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:27.410823 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:27.411230 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:27.411236 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:27.411239 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:27.411438 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:27.411985 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:31:27.411994 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:31:27.412339 (+ 345us) client_negotiation.cc:769] Sending connection context
0926 16:31:27.412625 (+ 286us) client_negotiation.cc:240] Negotiation successful
0926 16:31:27.413197 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":297,"threads_started":1}
I20260926 16:31:27.416270 12611 heartbeater.cc:347] Connected to a master server at 127.6.33.62:33939
I20260926 16:31:27.416594 12611 heartbeater.cc:464] Registering TS with master...
I20260926 16:31:27.417392 12611 heartbeater.cc:511] Master 127.6.33.62:33939 requested a full tablet report, sending...
I20260926 16:31:27.419456 12422 ts_manager.cc:194] Registered new tserver with Master: b1109a56871a442584d8a397a4c72542 (127.6.33.1:38529)
WARNING: no policy specified for oryx/127.6.33.2@KRBTEST.COM; defaulting to no policy
I20260926 16:31:27.421006 12422 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.33.1@KRBTEST.COM'} at 127.6.33.1:39519
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.33.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.33.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/krb5kdc/oryx_127.6.33.2.keytab.
Entry for principal oryx/127.6.33.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/krb5kdc/oryx_127.6.33.2.keytab.
I20260926 16:31:27.449497 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.2:0
--local_ip_for_outbound_sockets=127.6.33.2
--webserver_interface=127.6.33.2
--webserver_port=0
--tserver_master_addrs=127.6.33.62:33939
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/krb5kdc/oryx_127.6.33.2.keytab
--principal=oryx/127.6.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:42049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:31:27.789872 12617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:31:27.790258 12617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:31:27.790395 12617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:31:27.798900 12617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:31:27.799023 12617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:31:27.799140 12617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.2
I20260926 16:31:27.809329 12617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:42049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/krb5kdc/oryx_127.6.33.2.keytab
--principal=oryx/127.6.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.33.2
--webserver_port=0
--tserver_master_addrs=127.6.33.62:33939
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.2
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:31:27.810599 12617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:31:27.811894 12617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:31:27.825062 12625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:27.825080 12622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:27.825215 12623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:31:27.825960 12617 server_base.cc:1034] running on GCE node
I20260926 16:31:27.826592 12617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:31:27.827556 12617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:31:27.828840 12617 hybrid_clock.cc:648] HybridClock initialized: now 1790440287828779 us; error 43 us; skew 500 ppm
I20260926 16:31:27.829195 12617 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:31:27 dist-test-slave-khd3 krb5kdc[12377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440287, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:31:27.838557 12617 init.cc:377] Logged in from keytab as oryx/127.6.33.2@KRBTEST.COM (short username oryx)
I20260926 16:31:27.840085 12617 webserver.cc:460] Webserver started at http://127.6.33.2:38903/ using document root <none> and password file <none>
I20260926 16:31:27.840871 12617 fs_manager.cc:359] Metadata directory not provided
I20260926 16:31:27.841001 12617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:31:27.841310 12617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:31:27.844187 12617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-1/data/instance:
uuid: "d295b421f67a4d0e962b6c2365248448"
format_stamp: "Formatted at 2026-09-26 16:31:27 on dist-test-slave-khd3"
server_key: "b5e2bc1969ba42b61127c3789423627d"
server_key_iv: "0d7a30e9a7f3e639a13f6ef956538be3"
server_key_version: "encryptionkey@0"
I20260926 16:31:27.845002 12617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-1/wal/instance:
uuid: "d295b421f67a4d0e962b6c2365248448"
format_stamp: "Formatted at 2026-09-26 16:31:27 on dist-test-slave-khd3"
server_key: "b5e2bc1969ba42b61127c3789423627d"
server_key_iv: "0d7a30e9a7f3e639a13f6ef956538be3"
server_key_version: "encryptionkey@0"
I20260926 16:31:27.850196 12617 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260926 16:31:27.853811 12632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:27.855278 12617 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 16:31:27.855543 12617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-1/wal
uuid: "d295b421f67a4d0e962b6c2365248448"
format_stamp: "Formatted at 2026-09-26 16:31:27 on dist-test-slave-khd3"
server_key: "b5e2bc1969ba42b61127c3789423627d"
server_key_iv: "0d7a30e9a7f3e639a13f6ef956538be3"
server_key_version: "encryptionkey@0"
I20260926 16:31:27.855746 12617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:31:27.875499 12617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:31:27.879873 12617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:31:27.880163 12617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:31:27.881161 12617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:31:27.882493 12617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:31:27.882568 12617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:27.882694 12617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:31:27.882743 12617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:27.915665 12617 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.2:36957
I20260926 16:31:27.915766 12744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.2:36957 every 8 connection(s)
I20260926 16:31:27.917022 12617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
I20260926 16:31:27.920307 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 12617
I20260926 16:31:27.920513 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-1/wal/instance
I20260926 16:31:27.921087 6276 external_mini_cluster.cc:1428] Setting key 9fc896334390689c3b0de952be094857
Sep 26 16:31:27 dist-test-slave-khd3 krb5kdc[12377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440287, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.33.2@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM
I20260926 16:31:27.939152 12747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:27.919622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:54965 (local address 127.6.33.62:33939)
0926 16:31:27.920965 (+ 1343us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:27.920971 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:27.921106 (+ 135us) server_negotiation.cc:407] Connection header received
0926 16:31:27.923112 (+ 2006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:27.923117 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:27.923192 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:27.923301 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:27.924783 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:27.925902 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:27.927409 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:27.928160 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:27.928300 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:27.932169 (+ 3869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:27.932179 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:27.932187 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:27.932233 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:27.935748 (+ 3515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:27.936468 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:27.936476 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:27.936483 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:27.936641 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:27.937126 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:27.937134 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:27.937141 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:27.937566 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:27.937729 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:27.938436 (+ 707us) server_negotiation.cc:299] Negotiation successful
0926 16:31:27.938696 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1187,"thread_start_us":236,"threads_started":1}
I20260926 16:31:27.939993 12748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:27.920137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:33939 (local address 127.6.33.2:54965)
0926 16:31:27.920950 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:27.920995 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:27.922823 (+ 1828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:27.923549 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:27.923556 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:27.924047 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:27.924593 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:27.924609 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:27.926051 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:27.926055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:27.927268 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:27.927280 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:27.928387 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:27.928427 (+ 40us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:27.928639 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:27.929493 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:27.929532 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:27.931943 (+ 2411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:31:27.935897 (+ 3954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:27.935904 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:27.935917 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:27.936342 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:27.936748 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:27.936751 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:27.936753 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:27.936995 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:27.937691 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:31:27.937697 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:31:27.938035 (+ 338us) client_negotiation.cc:769] Sending connection context
0926 16:31:27.938259 (+ 224us) client_negotiation.cc:240] Negotiation successful
0926 16:31:27.938767 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":336,"threads_started":1}
I20260926 16:31:27.941540 12745 heartbeater.cc:347] Connected to a master server at 127.6.33.62:33939
I20260926 16:31:27.941851 12745 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for oryx/127.6.33.3@KRBTEST.COM; defaulting to no policy
I20260926 16:31:27.942615 12745 heartbeater.cc:511] Master 127.6.33.62:33939 requested a full tablet report, sending...
I20260926 16:31:27.944034 12422 ts_manager.cc:194] Registered new tserver with Master: d295b421f67a4d0e962b6c2365248448 (127.6.33.2:36957)
I20260926 16:31:27.944854 12422 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.33.2@KRBTEST.COM'} at 127.6.33.2:54965
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.33.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.33.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/krb5kdc/oryx_127.6.33.3.keytab.
Entry for principal oryx/127.6.33.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/krb5kdc/oryx_127.6.33.3.keytab.
I20260926 16:31:27.973033 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.3:0
--local_ip_for_outbound_sockets=127.6.33.3
--webserver_interface=127.6.33.3
--webserver_port=0
--tserver_master_addrs=127.6.33.62:33939
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/krb5kdc/oryx_127.6.33.3.keytab
--principal=oryx/127.6.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:42049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:31:28.301802 12751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:31:28.302163 12751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:31:28.302345 12751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:31:28.309890 12751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:31:28.310074 12751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:31:28.310272 12751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.3
I20260926 16:31:28.320662 12751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:42049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/krb5kdc/oryx_127.6.33.3.keytab
--principal=oryx/127.6.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.33.3
--webserver_port=0
--tserver_master_addrs=127.6.33.62:33939
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.3
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:31:28.322083 12751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:31:28.323535 12751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:31:28.337141 12756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:28.337186 12759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:28.337234 12757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:31:28.338140 12751 server_base.cc:1034] running on GCE node
I20260926 16:31:28.338835 12751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:31:28.339850 12751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:31:28.341071 12751 hybrid_clock.cc:648] HybridClock initialized: now 1790440288341025 us; error 43 us; skew 500 ppm
I20260926 16:31:28.341437 12751 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:31:28 dist-test-slave-khd3 krb5kdc[12377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440288, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:31:28.344570 12751 init.cc:377] Logged in from keytab as oryx/127.6.33.3@KRBTEST.COM (short username oryx)
I20260926 16:31:28.346129 12751 webserver.cc:460] Webserver started at http://127.6.33.3:43881/ using document root <none> and password file <none>
I20260926 16:31:28.346854 12751 fs_manager.cc:359] Metadata directory not provided
I20260926 16:31:28.346997 12751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:31:28.347294 12751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:31:28.350240 12751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-2/data/instance:
uuid: "93c3a12e5462468b875b5d0fbdd048de"
format_stamp: "Formatted at 2026-09-26 16:31:28 on dist-test-slave-khd3"
server_key: "a4baac71b7683e819f5630ea1e36527b"
server_key_iv: "b1be76d4259a98d630ce21a25a2fe934"
server_key_version: "encryptionkey@0"
I20260926 16:31:28.351022 12751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-2/wal/instance:
uuid: "93c3a12e5462468b875b5d0fbdd048de"
format_stamp: "Formatted at 2026-09-26 16:31:28 on dist-test-slave-khd3"
server_key: "a4baac71b7683e819f5630ea1e36527b"
server_key_iv: "b1be76d4259a98d630ce21a25a2fe934"
server_key_version: "encryptionkey@0"
I20260926 16:31:28.356472 12751 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 16:31:28.360203 12766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:28.361864 12751 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:31:28.362067 12751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-2/wal
uuid: "93c3a12e5462468b875b5d0fbdd048de"
format_stamp: "Formatted at 2026-09-26 16:31:28 on dist-test-slave-khd3"
server_key: "a4baac71b7683e819f5630ea1e36527b"
server_key_iv: "b1be76d4259a98d630ce21a25a2fe934"
server_key_version: "encryptionkey@0"
I20260926 16:31:28.362288 12751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:31:28.386484 12751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:31:28.390827 12751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:31:28.391196 12751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:31:28.392251 12751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:31:28.393679 12751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:31:28.393752 12751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:28.393896 12751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:31:28.393945 12751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:28.424134 12611 heartbeater.cc:503] Master 127.6.33.62:33939 was elected leader, sending a full tablet report...
I20260926 16:31:28.428447 12751 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.3:45149
I20260926 16:31:28.428504 12878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.3:45149 every 8 connection(s)
I20260926 16:31:28.429813 12751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
I20260926 16:31:28.434517 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 12751
I20260926 16:31:28.434705 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-2/wal/instance
I20260926 16:31:28.435118 6276 external_mini_cluster.cc:1428] Setting key 8e90865b9d4214abb57c1ac0341c7851
Sep 26 16:31:28 dist-test-slave-khd3 krb5kdc[12377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440288, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.33.3@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM
I20260926 16:31:28.449373 12747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:28.432441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.3:47105 (local address 127.6.33.62:33939)
0926 16:31:28.432610 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:28.432615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:28.433741 (+ 1126us) server_negotiation.cc:407] Connection header received
0926 16:31:28.435724 (+ 1983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:28.435732 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:28.435819 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:28.435945 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:28.437458 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:28.438185 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:28.439732 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:28.440149 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:28.440277 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:28.443623 (+ 3346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:28.443629 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:28.443634 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:28.443674 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:28.446206 (+ 2532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:28.447044 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:28.447050 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:28.447054 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:28.447191 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:28.447630 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:28.447635 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:28.447637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:28.447909 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:28.448060 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:28.448813 (+ 753us) server_negotiation.cc:299] Negotiation successful
0926 16:31:28.449057 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 16:31:28.450501 12881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:28.432889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:33939 (local address 127.6.33.3:47105)
0926 16:31:28.433575 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:28.433618 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:28.435455 (+ 1837us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:28.436173 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:28.436181 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:28.436659 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:28.437283 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:28.437299 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:28.438345 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:28.438353 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:28.439585 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:28.439597 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:28.440371 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:28.440378 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:28.440612 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:28.441284 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:28.441306 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:28.443404 (+ 2098us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:31:28.446465 (+ 3061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:28.446474 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:28.446492 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:28.446915 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:28.447304 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:28.447309 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:28.447313 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:28.447506 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:28.448042 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:31:28.448049 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:31:28.448397 (+ 348us) client_negotiation.cc:769] Sending connection context
0926 16:31:28.448663 (+ 266us) client_negotiation.cc:240] Negotiation successful
0926 16:31:28.449231 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":263,"threads_started":1}
I20260926 16:31:28.451794 12879 heartbeater.cc:347] Connected to a master server at 127.6.33.62:33939
I20260926 16:31:28.452114 12879 heartbeater.cc:464] Registering TS with master...
I20260926 16:31:28.452891 12879 heartbeater.cc:511] Master 127.6.33.62:33939 requested a full tablet report, sending...
I20260926 16:31:28.454285 12422 ts_manager.cc:194] Registered new tserver with Master: 93c3a12e5462468b875b5d0fbdd048de (127.6.33.3:45149)
I20260926 16:31:28.455035 12422 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.33.3@KRBTEST.COM'} at 127.6.33.3:47105
I20260926 16:31:28.462759 6276 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.6.33.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.33.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.33.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/krb5kdc/oryx_127.6.33.61.keytab.
Entry for principal oryx/127.6.33.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/krb5kdc/oryx_127.6.33.61.keytab.
I20260926 16:31:28.510746 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.61:37487
--webserver_interface=127.6.33.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/krb5kdc/oryx_127.6.33.61.keytab
--principal=oryx/127.6.33.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:42049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.33.62:33939,127.6.33.61:37487
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:31:28.835196 12884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:31:28.835543 12884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:31:28.835700 12884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:31:28.843087 12884 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:31:28.843245 12884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:31:28.843357 12884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:31:28.843464 12884 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:31:28.843541 12884 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:31:28.853392 12884 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:42049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.33.62:33939,127.6.33.61:37487
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/krb5kdc/oryx_127.6.33.61.keytab
--principal=oryx/127.6.33.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.61:37487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.33.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
W20260926 16:31:28.853812 12884 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.6.33.62:33939,127.6.33.61:37487'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260926 16:31:28.854914 12884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:31:28.856242 12884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:31:28.868925 12892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:28.868955 12889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:28.869014 12890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:31:28.869791 12884 server_base.cc:1034] running on GCE node
I20260926 16:31:28.870741 12884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:31:28.872277 12884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:31:28.873530 12884 hybrid_clock.cc:648] HybridClock initialized: now 1790440288873470 us; error 38 us; skew 500 ppm
I20260926 16:31:28.873951 12884 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:31:28 dist-test-slave-khd3 krb5kdc[12377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440288, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.33.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:31:28.877271 12884 init.cc:377] Logged in from keytab as oryx/127.6.33.61@KRBTEST.COM (short username oryx)
I20260926 16:31:28.878932 12884 webserver.cc:460] Webserver started at http://127.6.33.61:43901/ using document root <none> and password file <none>
I20260926 16:31:28.879724 12884 fs_manager.cc:359] Metadata directory not provided
I20260926 16:31:28.879846 12884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:31:28.880204 12884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:31:28.883312 12884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/master-1/data/instance:
uuid: "9d1dfe3b814a4549aebd349a75850ef4"
format_stamp: "Formatted at 2026-09-26 16:31:28 on dist-test-slave-khd3"
server_key: "a5e66bc771f8b1fa570d439196c44bf5"
server_key_iv: "1db6b1282d7ad376864d1fa47e082869"
server_key_version: "encryptionkey@0"
I20260926 16:31:28.884132 12884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/master-1/wal/instance:
uuid: "9d1dfe3b814a4549aebd349a75850ef4"
format_stamp: "Formatted at 2026-09-26 16:31:28 on dist-test-slave-khd3"
server_key: "a5e66bc771f8b1fa570d439196c44bf5"
server_key_iv: "1db6b1282d7ad376864d1fa47e082869"
server_key_version: "encryptionkey@0"
I20260926 16:31:28.889572 12884 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260926 16:31:28.892953 12899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:28.894332 12884 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260926 16:31:28.894512 12884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/master-1/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/master-1/wal
uuid: "9d1dfe3b814a4549aebd349a75850ef4"
format_stamp: "Formatted at 2026-09-26 16:31:28 on dist-test-slave-khd3"
server_key: "a5e66bc771f8b1fa570d439196c44bf5"
server_key_iv: "1db6b1282d7ad376864d1fa47e082869"
server_key_version: "encryptionkey@0"
I20260926 16:31:28.894724 12884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:31:28.908216 12884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:31:28.912614 12884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:31:28.912963 12884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:31:28.931999 12884 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.61:37487
I20260926 16:31:28.932016 12950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.61:37487 every 8 connection(s)
I20260926 16:31:28.933516 12884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/master-1/data/info.pb
I20260926 16:31:28.937387 12951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:31:28.940583 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 12884
I20260926 16:31:28.944746 12951 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.33.62" port: 33939 } has no permanent_uuid. Determining permanent_uuid...
I20260926 16:31:28.947309 12745 heartbeater.cc:503] Master 127.6.33.62:33939 was elected leader, sending a full tablet report...
Sep 26 16:31:28 dist-test-slave-khd3 krb5kdc[12377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.33.61@KRBTEST.COM
Sep 26 16:31:28 dist-test-slave-khd3 krb5kdc[12377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440288, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.33.61@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM
I20260926 16:31:28.962929 12747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:28.948114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50072 (local address 127.6.33.62:33939)
0926 16:31:28.948321 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:28.948328 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:28.949106 (+ 778us) server_negotiation.cc:407] Connection header received
0926 16:31:28.949366 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:28.949375 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:28.949431 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:28.949543 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:28.950553 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:28.951392 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:28.953006 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:28.953397 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:28.953537 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:28.956692 (+ 3155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:28.956700 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:28.956707 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:28.956748 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:28.960125 (+ 3377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:28.960856 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:28.960864 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:28.960868 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:28.960989 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:28.961344 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:28.961349 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:28.961352 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:28.961654 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:28.961804 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:28.962362 (+ 558us) server_negotiation.cc:299] Negotiation successful
0926 16:31:28.962573 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 16:31:28.963773 12955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:28.948096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:33939 (local address 127.0.0.1:50072)
0926 16:31:28.948984 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:28.949024 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:28.949217 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:28.949594 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:28.949601 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:28.950191 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:28.950413 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:28.950421 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:28.951540 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:28.951547 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:28.952852 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:28.952863 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:28.953616 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:28.953625 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:28.953785 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:28.954379 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:28.954399 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:28.956509 (+ 2110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:31:28.960248 (+ 3739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:28.960260 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:28.960273 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:28.960705 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:28.961094 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:28.961100 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:28.961104 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:28.961220 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:28.961787 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:31:28.961795 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:31:28.961978 (+ 183us) client_negotiation.cc:769] Sending connection context
0926 16:31:28.962243 (+ 265us) client_negotiation.cc:240] Negotiation successful
0926 16:31:28.962761 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"mutex_wait_us":147,"spinlock_wait_cycles":16768,"thread_start_us":457,"threads_started":1}
I20260926 16:31:28.963791 12954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:28.942783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57090 (local address 127.6.33.61:37487)
0926 16:31:28.943537 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:28.943549 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:28.943589 (+ 40us) server_negotiation.cc:407] Connection header received
0926 16:31:28.945250 (+ 1661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:28.945271 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:28.945687 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:28.946079 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:28.947833 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:28.949627 (+ 1794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:28.950869 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:28.951494 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:28.951693 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:28.955623 (+ 3930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:28.955631 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:28.955643 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:28.955669 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:28.959502 (+ 3833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:28.960299 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:28.960311 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:28.960313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:28.960484 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:28.960925 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:28.960930 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:28.960935 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:28.961457 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:28.961745 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:28.962167 (+ 422us) server_negotiation.cc:299] Negotiation successful
0926 16:31:28.962616 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":359,"threads_started":1}
I20260926 16:31:28.965911 12951 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.33.61" port: 37487 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 16:31:28 dist-test-slave-khd3 krb5kdc[12377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440288, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.33.61@KRBTEST.COM for oryx/127.6.33.61@KRBTEST.COM
I20260926 16:31:28.977615 12955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:28.966772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.61:37487 (local address 127.0.0.1:57098)
0926 16:31:28.967046 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:28.967064 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:28.967193 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:28.967455 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:28.967458 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:28.967746 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:28.967876 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:28.967883 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:28.968829 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:28.968833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:28.969665 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:28.969675 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:28.970279 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:28.970282 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:28.970405 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:28.971009 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:28.971026 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:28.972953 (+ 1927us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:31:28.975521 (+ 2568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:28.975525 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:28.975528 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:28.975857 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:28.976157 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:28.976160 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:28.976162 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:28.976274 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:28.976855 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:31:28.976858 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:31:28.976978 (+ 120us) client_negotiation.cc:769] Sending connection context
0926 16:31:28.977081 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 16:31:28.977271 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170}
I20260926 16:31:28.977695 12954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:28.966882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57098 (local address 127.6.33.61:37487)
0926 16:31:28.967048 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:28.967054 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:28.967174 (+ 120us) server_negotiation.cc:407] Connection header received
0926 16:31:28.967278 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:28.967281 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:28.967331 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:28.967404 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:28.967988 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:28.968667 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:28.969781 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:28.970163 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:28.970257 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:28.973122 (+ 2865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:28.973127 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:28.973130 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:28.973157 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:28.975399 (+ 2242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:28.975959 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:28.975963 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:28.975965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:28.976067 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:28.976366 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:28.976369 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:28.976371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:28.976701 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:28.976881 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:28.977157 (+ 276us) server_negotiation.cc:299] Negotiation successful
0926 16:31:28.977375 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":61}
I20260926 16:31:28.982244 12951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4: Bootstrap starting.
I20260926 16:31:28.985208 12951 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4: Neither blocks nor log segments found. Creating new log.
I20260926 16:31:28.986126 12951 log.cc:824] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4: Log is configured to *not* fsync() on all Append() calls
I20260926 16:31:28.988560 12951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4: No bootstrap required, opened a new log
I20260926 16:31:28.995301 12951 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a7973add6e4409dabfab1a4d9e6b175" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 33939 } } peers { permanent_uuid: "9d1dfe3b814a4549aebd349a75850ef4" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 37487 } }
I20260926 16:31:28.995600 12951 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:31:28.995669 12951 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d1dfe3b814a4549aebd349a75850ef4, State: Initialized, Role: FOLLOWER
I20260926 16:31:28.996142 12951 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a7973add6e4409dabfab1a4d9e6b175" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 33939 } } peers { permanent_uuid: "9d1dfe3b814a4549aebd349a75850ef4" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 37487 } }
I20260926 16:31:28.996932 12957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a7973add6e4409dabfab1a4d9e6b175" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 33939 } } peers { permanent_uuid: "9d1dfe3b814a4549aebd349a75850ef4" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 37487 } } }
I20260926 16:31:28.997262 12957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:31:28.998030 12951 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 16:31:29.006589 12968 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 16:31:29.006740 12968 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 16:31:29.008647 12884 master_runner.cc:428] Detected that this master 9d1dfe3b814a4549aebd349a75850ef4 is joining an existing cluster
I20260926 16:31:29.008790 12884 master_runner.cc:432] Initiating AddMaster RPC to add 127.6.33.61:37487
I20260926 16:31:29.029446 12747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:29.012491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50080 (local address 127.6.33.62:33939)
0926 16:31:29.012687 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:29.012693 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:29.013481 (+ 788us) server_negotiation.cc:407] Connection header received
0926 16:31:29.013702 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:29.013707 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:29.013777 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:29.013880 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:29.015000 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:29.015881 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:29.017817 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:29.018501 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:29.018600 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:29.021288 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:29.021297 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:29.021304 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:29.021346 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:29.025413 (+ 4067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:29.026242 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:29.026247 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:29.026251 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:29.026416 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:29.026900 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:29.026905 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:29.026909 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:29.027611 (+ 702us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:29.027751 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:29.028757 (+ 1006us) server_negotiation.cc:299] Negotiation successful
0926 16:31:29.029027 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 16:31:29.030588 12974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:29.012555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:33939 (local address 127.0.0.1:50080)
0926 16:31:29.013362 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:29.013385 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:29.013571 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:29.013902 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:29.013907 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:29.014455 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:29.014857 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:29.014867 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:29.016004 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:29.016010 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:29.017676 (+ 1666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:29.017685 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:29.018932 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:29.018941 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:29.019127 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:29.019904 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:29.019936 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:29.021067 (+ 1131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:31:29.025592 (+ 4525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:29.025597 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:29.025602 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:29.026106 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:29.026574 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:29.026578 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:29.026582 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:29.026760 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:29.027836 (+ 1076us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:31:29.027841 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:31:29.028064 (+ 223us) client_negotiation.cc:769] Sending connection context
0926 16:31:29.029719 (+ 1655us) client_negotiation.cc:240] Negotiation successful
0926 16:31:29.030049 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"mutex_wait_us":24,"thread_start_us":543,"threads_started":1}
I20260926 16:31:29.033581 12954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:29.012846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57114 (local address 127.6.33.61:37487)
0926 16:31:29.013039 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:29.013051 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:29.017410 (+ 4359us) server_negotiation.cc:407] Connection header received
0926 16:31:29.017627 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:29.017633 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:29.017697 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:29.017762 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:29.019374 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:29.020566 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:29.022877 (+ 2311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:29.023713 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:29.023818 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:29.026263 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:29.026268 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:29.026272 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:29.026306 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:29.029924 (+ 3618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:29.031438 (+ 1514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:29.031443 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:29.031446 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:29.031594 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:29.031978 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:29.031982 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:29.031985 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:29.032378 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:29.032550 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:29.033034 (+ 484us) server_negotiation.cc:299] Negotiation successful
0926 16:31:29.033241 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 16:31:29.034972 12975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:29.012781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.61:37487 (local address 127.0.0.1:57114)
0926 16:31:29.017322 (+ 4541us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:29.017344 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:29.017501 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:29.017937 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:29.017942 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:29.018712 (+ 770us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:29.019220 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:29.019233 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:29.020870 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:29.020876 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:29.022728 (+ 1852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:29.022744 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:29.024113 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:29.024119 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:29.024292 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:29.024920 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:29.024946 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:29.026104 (+ 1158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:31:29.030349 (+ 4245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:29.030354 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:29.030358 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:29.031309 (+ 951us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:29.031724 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:29.031728 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:29.031731 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:29.031871 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:29.032612 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:31:29.032616 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:31:29.032782 (+ 166us) client_negotiation.cc:769] Sending connection context
0926 16:31:29.034102 (+ 1320us) client_negotiation.cc:240] Negotiation successful
0926 16:31:29.034458 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4392,"mutex_wait_us":37,"thread_start_us":4603,"threads_started":1}
Sep 26 16:31:29 dist-test-slave-khd3 krb5kdc[12377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440286, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.33.62@KRBTEST.COM for oryx/127.6.33.61@KRBTEST.COM
WARNING: no policy specified for oryx/127.6.33.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.33.60@KRBTEST.COM" created.
I20260926 16:31:29.052065 12977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:29.037496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.61:37487 (local address 127.0.0.1:57122)
0926 16:31:29.038154 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:29.038199 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:29.038424 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:29.038871 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:29.038879 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:29.039530 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:29.039736 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:29.039744 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:29.040739 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:29.040746 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:29.041838 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:29.041848 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:29.042632 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:29.042637 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:29.042834 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:29.043566 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:29.043587 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:29.045749 (+ 2162us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:31:29.049116 (+ 3367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:29.049131 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:29.049139 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:29.049567 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:29.049992 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:29.049997 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:29.050001 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:29.050135 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:29.050782 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:31:29.050791 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:31:29.051023 (+ 232us) client_negotiation.cc:769] Sending connection context
0926 16:31:29.051264 (+ 241us) client_negotiation.cc:240] Negotiation successful
0926 16:31:29.051615 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":301,"threads_started":1}
I20260926 16:31:29.052145 12954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:29.037452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57122 (local address 127.6.33.61:37487)
0926 16:31:29.037607 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:29.037612 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:29.038278 (+ 666us) server_negotiation.cc:407] Connection header received
0926 16:31:29.038629 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:29.038634 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:29.038718 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:29.038822 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:29.039872 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:29.040610 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:29.041978 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:29.042468 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:29.042608 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:29.045982 (+ 3374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:29.045987 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:29.045990 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:29.046019 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:29.048979 (+ 2960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:29.049707 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:29.049712 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:29.049715 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:29.049877 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:29.050273 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:29.050283 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:29.050285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:29.050608 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:29.050778 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:29.051381 (+ 603us) server_negotiation.cc:299] Negotiation successful
0926 16:31:29.051652 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 16:31:29.058326 12422 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.6.33.61:37487: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "9d1dfe3b814a4549aebd349a75850ef4"
member_type: NON_VOTER
last_known_addr {
host: "127.6.33.61"
port: 37487
}
attrs {
promote: true
}
}
dest_uuid: "7a7973add6e4409dabfab1a4d9e6b175"
cas_config_opid_index: -1
I20260926 16:31:29.059581 12422 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a7973add6e4409dabfab1a4d9e6b175 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7a7973add6e4409dabfab1a4d9e6b175" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 33939 } } peers { permanent_uuid: "9d1dfe3b814a4549aebd349a75850ef4" member_type: NON_VOTER last_known_addr { host: "127.6.33.61" port: 37487 } attrs { promote: true } }
I20260926 16:31:29.073156 12980 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7a7973add6e4409dabfab1a4d9e6b175 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 9d1dfe3b814a4549aebd349a75850ef4 (127.6.33.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7a7973add6e4409dabfab1a4d9e6b175" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 33939 } } peers { permanent_uuid: "9d1dfe3b814a4549aebd349a75850ef4" member_type: NON_VOTER last_known_addr { host: "127.6.33.61" port: 37487 } attrs { promote: true } } }
I20260926 16:31:29.074968 12926 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:31:29.076534 12980 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.6.33.61:37487
I20260926 16:31:29.077047 12980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a7973add6e4409dabfab1a4d9e6b175 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7a7973add6e4409dabfab1a4d9e6b175" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7a7973add6e4409dabfab1a4d9e6b175" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 33939 } } peers { permanent_uuid: "9d1dfe3b814a4549aebd349a75850ef4" member_type: NON_VOTER last_known_addr { host: "127.6.33.61" port: 37487 } attrs { promote: true } } }
I20260926 16:31:29.077250 12980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a7973add6e4409dabfab1a4d9e6b175 [sys.catalog]: This master's current role is: LEADER
I20260926 16:31:29.077391 12884 master.cc:537] Master@127.6.33.61:37487 shutting down...
I20260926 16:31:29.080477 12926 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4 [term 1 FOLLOWER]: Refusing update from remote peer 7a7973add6e4409dabfab1a4d9e6b175: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260926 16:31:29.081730 12980 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7a7973add6e4409dabfab1a4d9e6b175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d1dfe3b814a4549aebd349a75850ef4" member_type: NON_VOTER last_known_addr { host: "127.6.33.61" port: 37487 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.33.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/krb5kdc/oryx_127.6.33.60.keytab.
Entry for principal oryx/127.6.33.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/krb5kdc/oryx_127.6.33.60.keytab.
I20260926 16:31:29.085500 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.60:42953
--webserver_interface=127.6.33.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/krb5kdc/oryx_127.6.33.60.keytab
--principal=oryx/127.6.33.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:42049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.33.62:33939,127.6.33.61:37487,127.6.33.60:42953
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:31:29.085708 12408 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260926 16:31:29.091449 12884 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:31:29.092001 12884 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:31:29.092320 12884 tablet_replica.cc:330] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4: stopping tablet replica
W20260926 16:31:29.098404 12408 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7a7973add6e4409dabfab1a4d9e6b175 -> Peer 9d1dfe3b814a4549aebd349a75850ef4 (127.6.33.61:37487): Couldn't send request to peer 9d1dfe3b814a4549aebd349a75850ef4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 16:31:29.102769 12884 master.cc:552] Master@127.6.33.61:37487 shutdown complete.
I20260926 16:31:29.102856 12884 master_runner.cc:305] Clearing existing system tablet
I20260926 16:31:29.103289 12884 fs_manager.cc:359] Metadata directory not provided
I20260926 16:31:29.103391 12884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:31:29.106276 12884 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260926 16:31:29.109211 12986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:29.109597 12884 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:31:29.109910 12884 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 16:31:29.110010 12884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/master-1/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/master-1/wal
uuid: "9d1dfe3b814a4549aebd349a75850ef4"
format_stamp: "Formatted at 2026-09-26 16:31:28 on dist-test-slave-khd3"
server_key: "a5e66bc771f8b1fa570d439196c44bf5"
server_key_iv: "1db6b1282d7ad376864d1fa47e082869"
server_key_version: "encryptionkey@0"
I20260926 16:31:29.116285 12884 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:31:29.123729 12884 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 16:31:29.123965 12884 log.cc:1196] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260926 16:31:29.124487 12884 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4: Deleting consensus metadata
I20260926 16:31:29.124819 12884 master_runner.cc:315] Copying system tablet from 127.6.33.62:33939
I20260926 16:31:29.129510 12884 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4: tablet copy: Beginning tablet copy session from remote peer at address 127.6.33.62:33939
I20260926 16:31:29.145849 12992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:29.130389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:33939 (local address 127.0.0.1:50092)
0926 16:31:29.131054 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:29.131082 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:29.131259 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:29.131670 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:29.131674 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:29.132260 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:29.132539 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:29.132552 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:29.134160 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:29.134166 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:29.135667 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:29.135682 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:29.136669 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:29.136675 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:29.136886 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:29.137633 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:29.137657 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:29.138814 (+ 1157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:31:29.142807 (+ 3993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:29.142813 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:29.142818 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:29.143354 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:29.143784 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:29.143789 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:29.143792 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:29.143995 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:29.144646 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:31:29.144651 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:31:29.144857 (+ 206us) client_negotiation.cc:769] Sending connection context
0926 16:31:29.145048 (+ 191us) client_negotiation.cc:240] Negotiation successful
0926 16:31:29.145371 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":386,"threads_started":1}
I20260926 16:31:29.145887 12747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:29.130404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50092 (local address 127.6.33.62:33939)
0926 16:31:29.130612 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:29.130621 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:29.131193 (+ 572us) server_negotiation.cc:407] Connection header received
0926 16:31:29.131410 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:29.131417 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:29.131504 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:29.131614 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:29.132860 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:29.133987 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:29.135824 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:29.136496 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:29.136626 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:29.139023 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:29.139030 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:29.139036 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:29.139076 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:29.142636 (+ 3560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:29.143483 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:29.143489 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:29.143493 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:29.143635 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:29.144135 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:29.144141 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:29.144144 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:29.144514 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:29.144660 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:29.145312 (+ 652us) server_negotiation.cc:299] Negotiation successful
0926 16:31:29.145532 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 16:31:29.147049 12442 tablet_copy_service.cc:140] P 7a7973add6e4409dabfab1a4d9e6b175: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 9d1dfe3b814a4549aebd349a75850ef4 ({username='oryx', principal='oryx/127.6.33.61@KRBTEST.COM'} at 127.0.0.1:50092)
I20260926 16:31:29.147344 12442 tablet_copy_service.cc:161] P 7a7973add6e4409dabfab1a4d9e6b175: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 9d1dfe3b814a4549aebd349a75850ef4 at {username='oryx', principal='oryx/127.6.33.61@KRBTEST.COM'} at 127.0.0.1:50092: session id = 9d1dfe3b814a4549aebd349a75850ef4-00000000000000000000000000000000
I20260926 16:31:29.152087 12442 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 7a7973add6e4409dabfab1a4d9e6b175: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 16:31:29.154474 12884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 16:31:29.161449 12884 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4: tablet copy: Starting download of 0 data blocks...
I20260926 16:31:29.161756 12884 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4: tablet copy: Starting download of 1 WAL segments...
I20260926 16:31:29.164640 12884 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 16:31:29.169165 12442 tablet_copy_service.cc:342] P 7a7973add6e4409dabfab1a4d9e6b175: Request end of tablet copy session 9d1dfe3b814a4549aebd349a75850ef4-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.6.33.61@KRBTEST.COM'} at 127.0.0.1:50092
I20260926 16:31:29.169409 12442 tablet_copy_service.cc:433] P 7a7973add6e4409dabfab1a4d9e6b175: ending tablet copy session 9d1dfe3b814a4549aebd349a75850ef4-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 9d1dfe3b814a4549aebd349a75850ef4
I20260926 16:31:29.181224 12884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:31:29.184195 12993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:29.184641 12994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:31:29.189569 12884 server_base.cc:1034] running on GCE node
W20260926 16:31:29.189924 12996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:31:29.190428 12884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:31:29.191222 12884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:31:29.192483 12884 hybrid_clock.cc:648] HybridClock initialized: now 1790440289192382 us; error 62 us; skew 500 ppm
I20260926 16:31:29.192809 12884 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:31:29 dist-test-slave-khd3 krb5kdc[12377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440289, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.33.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:31:29.195048 12884 init.cc:377] Logged in from keytab as oryx/127.6.33.61@KRBTEST.COM (short username oryx)
I20260926 16:31:29.196269 12884 webserver.cc:460] Webserver started at http://127.6.33.61:45703/ using document root <none> and password file <none>
I20260926 16:31:29.196633 12884 fs_manager.cc:359] Metadata directory not provided
I20260926 16:31:29.196709 12884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:31:29.199731 12884 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 16:31:29.201726 13003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:29.202196 12884 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 16:31:29.202392 12884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/master-1/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/master-1/wal
uuid: "9d1dfe3b814a4549aebd349a75850ef4"
format_stamp: "Formatted at 2026-09-26 16:31:28 on dist-test-slave-khd3"
server_key: "a5e66bc771f8b1fa570d439196c44bf5"
server_key_iv: "1db6b1282d7ad376864d1fa47e082869"
server_key_version: "encryptionkey@0"
I20260926 16:31:29.202633 12884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:31:29.224833 12884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:31:29.229154 12884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:31:29.247978 12884 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.61:37487
I20260926 16:31:29.248010 13054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.61:37487 every 8 connection(s)
I20260926 16:31:29.249123 12884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/master-1/data/info.pb
I20260926 16:31:29.254555 13055 sys_catalog.cc:263] Verifying existing consensus state
I20260926 16:31:29.258519 13055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4: Bootstrap starting.
I20260926 16:31:29.276154 13055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 16:31:29.276737 13055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4: Bootstrap complete.
I20260926 16:31:29.277805 13055 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7a7973add6e4409dabfab1a4d9e6b175" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 33939 } } peers { permanent_uuid: "9d1dfe3b814a4549aebd349a75850ef4" member_type: NON_VOTER last_known_addr { host: "127.6.33.61" port: 37487 } attrs { promote: true } }
I20260926 16:31:29.278056 13055 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9d1dfe3b814a4549aebd349a75850ef4, State: Initialized, Role: LEARNER
I20260926 16:31:29.278447 13055 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7a7973add6e4409dabfab1a4d9e6b175" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 33939 } } peers { permanent_uuid: "9d1dfe3b814a4549aebd349a75850ef4" member_type: NON_VOTER last_known_addr { host: "127.6.33.61" port: 37487 } attrs { promote: true } }
I20260926 16:31:29.279268 13058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7a7973add6e4409dabfab1a4d9e6b175" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 33939 } } peers { permanent_uuid: "9d1dfe3b814a4549aebd349a75850ef4" member_type: NON_VOTER last_known_addr { host: "127.6.33.61" port: 37487 } attrs { promote: true } } }
I20260926 16:31:29.279691 13058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4 [sys.catalog]: This master's current role is: LEARNER
I20260926 16:31:29.279546 13055 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:31:29.290375 13069 catalog_manager.cc:1259] Loaded cluster ID: cb13d77c4ee3430abf37211ff83e1ee5
I20260926 16:31:29.290517 13069 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4: loading cluster ID for follower catalog manager: success
I20260926 16:31:29.294117 13069 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4: acquiring CA information for follower catalog manager: success
I20260926 16:31:29.297056 13069 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 16:31:29.457626 12879 heartbeater.cc:503] Master 127.6.33.62:33939 was elected leader, sending a full tablet report...
W20260926 16:31:29.475284 12984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:31:29.475576 12984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:31:29.475665 12984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:31:29.483619 12984 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:31:29.483753 12984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:31:29.483829 12984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:31:29.483915 12984 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:31:29.483973 12984 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:31:29.492035 12977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:29.485285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.61:37487 (local address 127.0.0.1:57126)
0926 16:31:29.485534 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:29.485556 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:29.485696 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:29.486530 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:29.486534 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:29.486564 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:31:29.486715 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:29.486723 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:29.488617 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:29.488621 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:29.489885 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:29.489899 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:29.491497 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:29.491501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:29.491649 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:29.491688 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 16:31:29.491747 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 16:31:29.491842 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 16:31:29.492235 13073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:29.485514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57126 (local address 127.6.33.61:37487)
0926 16:31:29.486124 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:29.486130 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:29.486152 (+ 22us) server_negotiation.cc:407] Connection header received
0926 16:31:29.486222 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:29.486228 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:29.486335 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:29.486467 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:31:29.486862 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:29.488474 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:29.490029 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:29.491344 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:29.491472 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:29.491717 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:29.491785 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 16:31:29.491910 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":325,"threads_started":1}
I20260926 16:31:29.493234 13030 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260926 16:31:29.493881 13058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a7973add6e4409dabfab1a4d9e6b175. Latest consensus state: current_term: 1 leader_uuid: "7a7973add6e4409dabfab1a4d9e6b175" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7a7973add6e4409dabfab1a4d9e6b175" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 33939 } } peers { permanent_uuid: "9d1dfe3b814a4549aebd349a75850ef4" member_type: NON_VOTER last_known_addr { host: "127.6.33.61" port: 37487 } attrs { promote: true } } }
I20260926 16:31:29.494167 13058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4 [sys.catalog]: This master's current role is: LEARNER
I20260926 16:31:29.496688 12984 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:42049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.33.62:33939,127.6.33.61:37487,127.6.33.60:42953
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/krb5kdc/oryx_127.6.33.60.keytab
--principal=oryx/127.6.33.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.60:42953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.33.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:31:29.498081 12984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:31:29.499387 12984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:31:29.511261 13078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:29.511329 13075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:29.511329 13076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:31:29.512290 12984 server_base.cc:1034] running on GCE node
I20260926 16:31:29.513187 12984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:31:29.514739 12984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:31:29.515995 12984 hybrid_clock.cc:648] HybridClock initialized: now 1790440289515955 us; error 58 us; skew 500 ppm
I20260926 16:31:29.516337 12984 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:31:29 dist-test-slave-khd3 krb5kdc[12377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440289, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.33.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:31:29.519685 12984 init.cc:377] Logged in from keytab as oryx/127.6.33.60@KRBTEST.COM (short username oryx)
I20260926 16:31:29.521298 12984 webserver.cc:460] Webserver started at http://127.6.33.60:46515/ using document root <none> and password file <none>
I20260926 16:31:29.521999 12984 fs_manager.cc:359] Metadata directory not provided
I20260926 16:31:29.522136 12984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:31:29.522437 12984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:31:29.525393 12984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/master-2/data/instance:
uuid: "8b9d3065bcd44983a97ac74155813cdc"
format_stamp: "Formatted at 2026-09-26 16:31:29 on dist-test-slave-khd3"
server_key: "1ac152f8d2cf6aad488d2eb1cb3e9dd5"
server_key_iv: "657d38907c7309f67318a1ea117539b3"
server_key_version: "encryptionkey@0"
I20260926 16:31:29.526149 12984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/master-2/wal/instance:
uuid: "8b9d3065bcd44983a97ac74155813cdc"
format_stamp: "Formatted at 2026-09-26 16:31:29 on dist-test-slave-khd3"
server_key: "1ac152f8d2cf6aad488d2eb1cb3e9dd5"
server_key_iv: "657d38907c7309f67318a1ea117539b3"
server_key_version: "encryptionkey@0"
I20260926 16:31:29.531328 12984 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 16:31:29.534682 13085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:29.536038 12984 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 16:31:29.536216 12984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/master-2/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/master-2/wal
uuid: "8b9d3065bcd44983a97ac74155813cdc"
format_stamp: "Formatted at 2026-09-26 16:31:29 on dist-test-slave-khd3"
server_key: "1ac152f8d2cf6aad488d2eb1cb3e9dd5"
server_key_iv: "657d38907c7309f67318a1ea117539b3"
server_key_version: "encryptionkey@0"
I20260926 16:31:29.536399 12984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:31:29.578279 12984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:31:29.583086 12984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:31:29.583395 12984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:31:29.601755 12984 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.60:42953
I20260926 16:31:29.601787 13136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.60:42953 every 8 connection(s)
I20260926 16:31:29.603181 12984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/master-2/data/info.pb
I20260926 16:31:29.606277 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 12984
I20260926 16:31:29.607299 13137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:31:29.617542 13137 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.33.62" port: 33939 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 16:31:29 dist-test-slave-khd3 krb5kdc[12377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.33.60@KRBTEST.COM
Sep 26 16:31:29 dist-test-slave-khd3 krb5kdc[12377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440289, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.33.60@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM
I20260926 16:31:29.632717 13140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:29.608521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46168 (local address 127.6.33.60:42953)
0926 16:31:29.609337 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:29.609352 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:29.609400 (+ 48us) server_negotiation.cc:407] Connection header received
0926 16:31:29.610996 (+ 1596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:29.611023 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:29.611547 (+ 524us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:29.612020 (+ 473us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:29.613382 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:29.615445 (+ 2063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:29.616716 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:29.617769 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:29.617990 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:29.623575 (+ 5585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:29.623585 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:29.623598 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:29.623639 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:29.627322 (+ 3683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:29.628139 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:29.628148 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:29.628157 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:29.628356 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:29.628750 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:29.628774 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:29.628778 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:29.629494 (+ 716us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:29.629759 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:29.631028 (+ 1269us) server_negotiation.cc:299] Negotiation successful
0926 16:31:29.631439 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":351,"threads_started":1}
I20260926 16:31:29.638895 12747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:29.620925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50096 (local address 127.6.33.62:33939)
0926 16:31:29.621076 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:29.621080 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:29.621561 (+ 481us) server_negotiation.cc:407] Connection header received
0926 16:31:29.621781 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:29.621788 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:29.621890 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:29.621987 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:29.623575 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:29.624580 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:29.626981 (+ 2401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:29.627591 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:29.627681 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:29.631990 (+ 4309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:29.631999 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:29.632007 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:29.632049 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:29.635654 (+ 3605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:29.636532 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:29.636538 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:29.636542 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:29.636689 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:29.637265 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:29.637270 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:29.637274 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:29.637671 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:29.637852 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:29.638341 (+ 489us) server_negotiation.cc:299] Negotiation successful
0926 16:31:29.638554 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 16:31:29.639219 13141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:29.620659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:33939 (local address 127.0.0.1:50096)
0926 16:31:29.621401 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:29.621452 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:29.621610 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:29.622101 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:29.622111 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:29.623093 (+ 982us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:29.623396 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:29.623407 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:29.624797 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:29.624802 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:29.626827 (+ 2025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:29.626840 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:29.627932 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:29.627938 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:29.628122 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:29.628872 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:29.628894 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:29.631751 (+ 2857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:31:29.635802 (+ 4051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:29.635810 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:29.635816 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:29.636377 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:29.636867 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:29.636873 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:29.636882 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:29.637081 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:29.637827 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:31:29.637834 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:31:29.638011 (+ 177us) client_negotiation.cc:769] Sending connection context
0926 16:31:29.638215 (+ 204us) client_negotiation.cc:240] Negotiation successful
0926 16:31:29.638554 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"spinlock_wait_cycles":18816,"thread_start_us":416,"threads_started":1}
I20260926 16:31:29.641065 13137 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.33.61" port: 37487 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 16:31:29 dist-test-slave-khd3 krb5kdc[12377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440289, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.33.60@KRBTEST.COM for oryx/127.6.33.61@KRBTEST.COM
I20260926 16:31:29.653090 13141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:29.641859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.61:37487 (local address 127.0.0.1:57136)
0926 16:31:29.642045 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:29.642065 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:29.642163 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:29.642497 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:29.642502 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:29.642841 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:29.642971 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:29.642979 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:29.643965 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:29.643968 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:29.644915 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:29.644927 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:29.645567 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:29.645571 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:29.645700 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:29.646269 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:29.646285 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:29.648148 (+ 1863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:31:29.650941 (+ 2793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:29.650945 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:29.650948 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:29.651278 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:29.651796 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:29.651799 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:29.651801 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:29.651918 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:29.652386 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:31:29.652390 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:31:29.652511 (+ 121us) client_negotiation.cc:769] Sending connection context
0926 16:31:29.652614 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 16:31:29.652876 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 16:31:29.653155 13073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:29.641931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57136 (local address 127.6.33.61:37487)
0926 16:31:29.642159 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:29.642165 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:29.642183 (+ 18us) server_negotiation.cc:407] Connection header received
0926 16:31:29.642268 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:29.642274 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:29.642355 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:29.642458 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:29.643103 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:29.643849 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:29.645032 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:29.645428 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:29.645554 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:29.648346 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:29.648351 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:29.648355 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:29.648384 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:29.650793 (+ 2409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:29.651559 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:29.651563 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:29.651565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:29.651687 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:29.652019 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:29.652022 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:29.652024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:29.652285 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:29.652410 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:29.652700 (+ 290us) server_negotiation.cc:299] Negotiation successful
0926 16:31:29.652934 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260926 16:31:29.654019 13137 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.33.60" port: 42953 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 16:31:29 dist-test-slave-khd3 krb5kdc[12377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440289, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.33.60@KRBTEST.COM for oryx/127.6.33.60@KRBTEST.COM
I20260926 16:31:29.666079 13141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:29.654782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:42953 (local address 127.0.0.1:46170)
0926 16:31:29.654945 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:29.654965 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:29.655092 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:29.655417 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:29.655422 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:29.655738 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:29.655863 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:29.655870 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:29.656805 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:29.656811 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:29.657624 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:29.657634 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:29.658310 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:29.658322 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:29.658512 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:29.659021 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:29.659035 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:29.661136 (+ 2101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:31:29.663774 (+ 2638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:29.663778 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:29.663782 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:29.664123 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:29.664475 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:29.664481 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:29.664485 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:29.664621 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:29.665114 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:31:29.665118 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:31:29.665241 (+ 123us) client_negotiation.cc:769] Sending connection context
0926 16:31:29.665371 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 16:31:29.665638 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 16:31:29.666111 13140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:29.654748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46170 (local address 127.6.33.60:42953)
0926 16:31:29.654946 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:29.654953 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:29.655035 (+ 82us) server_negotiation.cc:407] Connection header received
0926 16:31:29.655192 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:29.655197 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:29.655260 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:29.655352 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:29.655984 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:29.656667 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:29.657753 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:29.658154 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:29.658295 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:29.661296 (+ 3001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:29.661301 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:29.661306 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:29.661341 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:29.663614 (+ 2273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:29.664231 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:29.664239 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:29.664243 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:29.664367 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:29.664718 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:29.664726 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:29.664729 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:29.665008 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:29.665149 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:29.665459 (+ 310us) server_negotiation.cc:299] Negotiation successful
0926 16:31:29.665776 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":119,"server-negotiator.queue_time_us":81}
I20260926 16:31:29.670972 13137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b9d3065bcd44983a97ac74155813cdc: Bootstrap starting.
I20260926 16:31:29.674047 13137 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b9d3065bcd44983a97ac74155813cdc: Neither blocks nor log segments found. Creating new log.
I20260926 16:31:29.675040 13137 log.cc:824] T 00000000000000000000000000000000 P 8b9d3065bcd44983a97ac74155813cdc: Log is configured to *not* fsync() on all Append() calls
I20260926 16:31:29.677651 13137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b9d3065bcd44983a97ac74155813cdc: No bootstrap required, opened a new log
I20260926 16:31:29.685015 13137 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b9d3065bcd44983a97ac74155813cdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a7973add6e4409dabfab1a4d9e6b175" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 33939 } } peers { permanent_uuid: "9d1dfe3b814a4549aebd349a75850ef4" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 37487 } } peers { permanent_uuid: "8b9d3065bcd44983a97ac74155813cdc" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 42953 } }
I20260926 16:31:29.685343 13137 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b9d3065bcd44983a97ac74155813cdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:31:29.685441 13137 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b9d3065bcd44983a97ac74155813cdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b9d3065bcd44983a97ac74155813cdc, State: Initialized, Role: FOLLOWER
I20260926 16:31:29.685979 13137 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b9d3065bcd44983a97ac74155813cdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a7973add6e4409dabfab1a4d9e6b175" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 33939 } } peers { permanent_uuid: "9d1dfe3b814a4549aebd349a75850ef4" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 37487 } } peers { permanent_uuid: "8b9d3065bcd44983a97ac74155813cdc" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 42953 } }
I20260926 16:31:29.686904 13143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b9d3065bcd44983a97ac74155813cdc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a7973add6e4409dabfab1a4d9e6b175" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 33939 } } peers { permanent_uuid: "9d1dfe3b814a4549aebd349a75850ef4" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 37487 } } peers { permanent_uuid: "8b9d3065bcd44983a97ac74155813cdc" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 42953 } } }
I20260926 16:31:29.687323 13143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b9d3065bcd44983a97ac74155813cdc [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:31:29.688107 13137 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b9d3065bcd44983a97ac74155813cdc [sys.catalog]: configured and running, proceeding with master startup.
W20260926 16:31:29.697302 13154 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8b9d3065bcd44983a97ac74155813cdc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 16:31:29.697464 13154 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 16:31:29.701352 12984 master_runner.cc:428] Detected that this master 8b9d3065bcd44983a97ac74155813cdc is joining an existing cluster
I20260926 16:31:29.701501 12984 master_runner.cc:432] Initiating AddMaster RPC to add 127.6.33.60:42953
I20260926 16:31:29.730861 13140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:29.706739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46178 (local address 127.6.33.60:42953)
0926 16:31:29.707015 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:29.707025 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:29.713549 (+ 6524us) server_negotiation.cc:407] Connection header received
0926 16:31:29.714105 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:29.714111 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:29.714177 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:29.714242 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:29.716071 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:29.717277 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:29.719608 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:29.720298 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:29.720394 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:29.723468 (+ 3074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:29.723474 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:29.723478 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:29.723515 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:29.727766 (+ 4251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:29.728491 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:29.728496 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:29.728499 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:29.728681 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:29.729117 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:29.729121 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:29.729123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:29.729553 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:29.729676 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:29.730254 (+ 578us) server_negotiation.cc:299] Negotiation successful
0926 16:31:29.730470 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 16:31:29.730834 13073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:29.706249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57138 (local address 127.6.33.61:37487)
0926 16:31:29.706486 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:29.706492 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:29.714093 (+ 7601us) server_negotiation.cc:407] Connection header received
0926 16:31:29.714356 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:29.714363 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:29.714460 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:29.714533 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:29.716144 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:29.717276 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:29.719622 (+ 2346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:29.720298 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:29.720421 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:29.723484 (+ 3063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:29.723489 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:29.723493 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:29.723530 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:29.727514 (+ 3984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:29.728359 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:29.728365 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:29.728379 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:29.728529 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:29.729082 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:29.729090 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:29.729096 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:29.729459 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:29.729584 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:29.730307 (+ 723us) server_negotiation.cc:299] Negotiation successful
0926 16:31:29.730531 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260926 16:31:29.732277 13160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:29.706281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.61:37487 (local address 127.0.0.1:57138)
0926 16:31:29.713995 (+ 7714us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:29.714014 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:29.714173 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:29.714771 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:29.714777 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:29.715567 (+ 790us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:29.715979 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:29.715992 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:29.717467 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:29.717473 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:29.719454 (+ 1981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:29.719472 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:29.720727 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:29.720735 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:29.720988 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:29.722011 (+ 1023us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:29.722032 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:29.723279 (+ 1247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:31:29.727673 (+ 4394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:29.727680 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:29.727687 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:29.728222 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:29.728687 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:29.728691 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:29.728694 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:29.728892 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:29.729676 (+ 784us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:31:29.729681 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:31:29.729973 (+ 292us) client_negotiation.cc:769] Sending connection context
0926 16:31:29.731162 (+ 1189us) client_negotiation.cc:240] Negotiation successful
0926 16:31:29.731492 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7567,"mutex_wait_us":255,"spinlock_wait_cycles":61824,"thread_start_us":7292,"threads_started":1}
I20260926 16:31:29.732324 13161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:29.706281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:42953 (local address 127.0.0.1:46178)
0926 16:31:29.713411 (+ 7130us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:29.713441 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:29.713948 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:29.714444 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:29.714448 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:29.715355 (+ 907us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:29.715894 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:29.715907 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:29.717477 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:29.717481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:29.719454 (+ 1973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:29.719472 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:29.720745 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:29.720750 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:29.721000 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:29.721889 (+ 889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:29.721923 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:29.723281 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:31:29.727910 (+ 4629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:29.727914 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:29.727918 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:29.728380 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:29.728856 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:29.728860 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:29.728864 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:29.729021 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:29.729744 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:31:29.729748 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:31:29.730005 (+ 257us) client_negotiation.cc:769] Sending connection context
0926 16:31:29.731597 (+ 1592us) client_negotiation.cc:240] Negotiation successful
0926 16:31:29.731911 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6983,"mutex_wait_us":217,"thread_start_us":6814,"threads_started":1}
I20260926 16:31:29.738919 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 12483
I20260926 16:31:29.740584 12747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:29.707150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50106 (local address 127.6.33.62:33939)
0926 16:31:29.707377 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:29.707385 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:29.725159 (+ 17774us) server_negotiation.cc:407] Connection header received
0926 16:31:29.725356 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:29.725361 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:29.725440 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:29.725537 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:29.727149 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:29.728282 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:29.730600 (+ 2318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:29.731277 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:29.731390 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:29.734087 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:29.734096 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:29.734104 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:29.734143 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:29.737575 (+ 3432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:29.738369 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:29.738375 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:29.738379 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:29.738534 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:29.739012 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:29.739018 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:29.739021 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:29.739390 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:29.739546 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:29.740045 (+ 499us) server_negotiation.cc:299] Negotiation successful
0926 16:31:29.740298 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 16:31:29.741508 13162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:29.707715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:33939 (local address 127.0.0.1:50106)
0926 16:31:29.725041 (+ 17326us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:29.725059 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:29.725211 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:29.725717 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:29.725721 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:29.726539 (+ 818us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:29.726961 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:29.726973 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:29.728477 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:29.728483 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:29.730429 (+ 1946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:29.730445 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:29.731688 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:29.731695 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:29.731906 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:29.732562 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:29.732584 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:29.733896 (+ 1312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:31:29.737707 (+ 3811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:29.737715 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:29.737721 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:29.738210 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:29.738672 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:29.738677 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:29.738681 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:29.738872 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:29.739513 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:31:29.739518 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:31:29.739738 (+ 220us) client_negotiation.cc:769] Sending connection context
0926 16:31:29.739905 (+ 167us) client_negotiation.cc:240] Negotiation successful
0926 16:31:29.740227 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17181,"thread_start_us":16945,"threads_started":1}
Sep 26 16:31:29 dist-test-slave-khd3 krb5kdc[12377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.33.1@KRBTEST.COM
Sep 26 16:31:29 dist-test-slave-khd3 krb5kdc[12377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440286, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.33.62@KRBTEST.COM for oryx/127.6.33.60@KRBTEST.COM
I20260926 16:31:29.761107 12977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:29.747206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:42953 (local address 127.0.0.1:46192)
0926 16:31:29.747476 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:29.747495 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:29.747592 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:29.748178 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:29.748185 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:29.748637 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:29.748866 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:29.748888 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:29.750288 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:29.750292 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:29.751152 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:29.751163 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:29.751900 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:29.751906 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:29.752068 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:29.752623 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:29.752639 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:29.755136 (+ 2497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:31:29.758382 (+ 3246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:29.758390 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:29.758394 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:29.758744 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:29.759258 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:29.759263 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:29.759266 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:29.759418 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:29.760029 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:31:29.760034 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:31:29.760217 (+ 183us) client_negotiation.cc:769] Sending connection context
0926 16:31:29.760414 (+ 197us) client_negotiation.cc:240] Negotiation successful
0926 16:31:29.760703 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157}
I20260926 16:31:29.761137 13140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:29.747286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46192 (local address 127.6.33.60:42953)
0926 16:31:29.747784 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:29.747789 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:29.747804 (+ 15us) server_negotiation.cc:407] Connection header received
0926 16:31:29.747914 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:29.747919 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:29.748027 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:29.748134 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:29.749037 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:29.750058 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:29.751288 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:29.751736 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:29.751883 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:29.755367 (+ 3484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:29.755372 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:29.755375 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:29.755407 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:29.758223 (+ 2816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:29.758943 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:29.758946 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:29.758949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:29.759129 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:29.759531 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:29.759538 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:29.759541 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:29.759840 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:29.759990 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:29.760539 (+ 549us) server_negotiation.cc:299] Negotiation successful
0926 16:31:29.760778 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392}
I20260926 16:31:29.762676 13163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:29.739943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39902 (local address 127.6.33.1:38529)
0926 16:31:29.742912 (+ 2969us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:29.742919 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:29.742945 (+ 26us) server_negotiation.cc:407] Connection header received
0926 16:31:29.742999 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:29.743005 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:29.743319 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:29.743498 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:29.744903 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:29.746494 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:29.748284 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:29.748949 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:29.749629 (+ 680us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:29.754809 (+ 5180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:29.754826 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:29.754846 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:29.754900 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:29.758947 (+ 4047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:29.759835 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:29.759846 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:29.759849 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:29.760061 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:29.760514 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:29.760520 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:29.760523 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:29.760991 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:29.761267 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:29.761865 (+ 598us) server_negotiation.cc:299] Negotiation successful
0926 16:31:29.762126 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2795,"thread_start_us":2619,"threads_started":1}
I20260926 16:31:29.762887 12422 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.6.33.60:42953: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "8b9d3065bcd44983a97ac74155813cdc"
member_type: NON_VOTER
last_known_addr {
host: "127.6.33.60"
port: 42953
}
attrs {
promote: true
}
}
dest_uuid: "7a7973add6e4409dabfab1a4d9e6b175"
cas_config_opid_index: 5
I20260926 16:31:29.763883 12422 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a7973add6e4409dabfab1a4d9e6b175 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7a7973add6e4409dabfab1a4d9e6b175" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 33939 } } peers { permanent_uuid: "9d1dfe3b814a4549aebd349a75850ef4" member_type: NON_VOTER last_known_addr { host: "127.6.33.61" port: 37487 } attrs { promote: true } } peers { permanent_uuid: "8b9d3065bcd44983a97ac74155813cdc" member_type: NON_VOTER last_known_addr { host: "127.6.33.60" port: 42953 } attrs { promote: true } }
I20260926 16:31:29.764259 12606 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:31:29.765525 12980 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7a7973add6e4409dabfab1a4d9e6b175 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 8b9d3065bcd44983a97ac74155813cdc (127.6.33.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7a7973add6e4409dabfab1a4d9e6b175" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 33939 } } peers { permanent_uuid: "9d1dfe3b814a4549aebd349a75850ef4" member_type: NON_VOTER last_known_addr { host: "127.6.33.61" port: 37487 } attrs { promote: true } } peers { permanent_uuid: "8b9d3065bcd44983a97ac74155813cdc" member_type: NON_VOTER last_known_addr { host: "127.6.33.60" port: 42953 } attrs { promote: true } } }
I20260926 16:31:29.779839 13112 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b9d3065bcd44983a97ac74155813cdc [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:31:29.793357 13030 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4 [term 1 LEARNER]: Refusing update from remote peer 7a7973add6e4409dabfab1a4d9e6b175: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 16:31:29.794730 13112 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8b9d3065bcd44983a97ac74155813cdc [term 1 FOLLOWER]: Refusing update from remote peer 7a7973add6e4409dabfab1a4d9e6b175: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 16:31:29.801857 12980 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.6.33.60:42953
I20260926 16:31:29.802619 12984 master.cc:537] Master@127.6.33.60:42953 shutting down...
I20260926 16:31:29.803534 13165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a7973add6e4409dabfab1a4d9e6b175 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7a7973add6e4409dabfab1a4d9e6b175" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7a7973add6e4409dabfab1a4d9e6b175" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 33939 } } peers { permanent_uuid: "9d1dfe3b814a4549aebd349a75850ef4" member_type: NON_VOTER last_known_addr { host: "127.6.33.61" port: 37487 } attrs { promote: true } } peers { permanent_uuid: "8b9d3065bcd44983a97ac74155813cdc" member_type: NON_VOTER last_known_addr { host: "127.6.33.60" port: 42953 } attrs { promote: true } } }
I20260926 16:31:29.804015 13165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a7973add6e4409dabfab1a4d9e6b175 [sys.catalog]: This master's current role is: LEADER
I20260926 16:31:29.804447 13169 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7a7973add6e4409dabfab1a4d9e6b175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d1dfe3b814a4549aebd349a75850ef4" member_type: NON_VOTER last_known_addr { host: "127.6.33.61" port: 37487 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.003s
I20260926 16:31:29.811548 13166 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7a7973add6e4409dabfab1a4d9e6b175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b9d3065bcd44983a97ac74155813cdc" member_type: NON_VOTER last_known_addr { host: "127.6.33.60" port: 42953 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:31:29.816463 13030 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 8b9d3065bcd44983a97ac74155813cdc (127.6.33.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7a7973add6e4409dabfab1a4d9e6b175" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 33939 } } peers { permanent_uuid: "9d1dfe3b814a4549aebd349a75850ef4" member_type: NON_VOTER last_known_addr { host: "127.6.33.61" port: 37487 } attrs { promote: true } } peers { permanent_uuid: "8b9d3065bcd44983a97ac74155813cdc" member_type: NON_VOTER last_known_addr { host: "127.6.33.60" port: 42953 } attrs { promote: true } } }
I20260926 16:31:29.820603 13058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7a7973add6e4409dabfab1a4d9e6b175" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7a7973add6e4409dabfab1a4d9e6b175" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 33939 } } peers { permanent_uuid: "9d1dfe3b814a4549aebd349a75850ef4" member_type: NON_VOTER last_known_addr { host: "127.6.33.61" port: 37487 } attrs { promote: true } } peers { permanent_uuid: "8b9d3065bcd44983a97ac74155813cdc" member_type: NON_VOTER last_known_addr { host: "127.6.33.60" port: 42953 } attrs { promote: true } } }
I20260926 16:31:29.820906 13058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4 [sys.catalog]: This master's current role is: LEARNER
I20260926 16:31:29.825248 13167 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 7a7973add6e4409dabfab1a4d9e6b175: attempting to promote NON_VOTER 9d1dfe3b814a4549aebd349a75850ef4 to VOTER
I20260926 16:31:29.826071 13167 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a7973add6e4409dabfab1a4d9e6b175 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7a7973add6e4409dabfab1a4d9e6b175" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 33939 } } peers { permanent_uuid: "9d1dfe3b814a4549aebd349a75850ef4" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 37487 } attrs { promote: false } } peers { permanent_uuid: "8b9d3065bcd44983a97ac74155813cdc" member_type: NON_VOTER last_known_addr { host: "127.6.33.60" port: 42953 } attrs { promote: true } }
I20260926 16:31:29.828693 13030 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4 [term 1 LEARNER]: Refusing update from remote peer 7a7973add6e4409dabfab1a4d9e6b175: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260926 16:31:29.834954 13167 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7a7973add6e4409dabfab1a4d9e6b175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d1dfe3b814a4549aebd349a75850ef4" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 37487 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260926 16:31:29.837997 13091 tablet.cc:2369] T 00000000000000000000000000000000 P 8b9d3065bcd44983a97ac74155813cdc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:31:29.840271 12984 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 8b9d3065bcd44983a97ac74155813cdc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:31:29.840862 12984 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 8b9d3065bcd44983a97ac74155813cdc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:31:29.841096 12984 tablet_replica.cc:330] T 00000000000000000000000000000000 P 8b9d3065bcd44983a97ac74155813cdc: stopping tablet replica
I20260926 16:31:29.845747 13167 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7a7973add6e4409dabfab1a4d9e6b175 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9d1dfe3b814a4549aebd349a75850ef4 (127.6.33.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7a7973add6e4409dabfab1a4d9e6b175" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 33939 } } peers { permanent_uuid: "9d1dfe3b814a4549aebd349a75850ef4" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 37487 } attrs { promote: false } } peers { permanent_uuid: "8b9d3065bcd44983a97ac74155813cdc" member_type: NON_VOTER last_known_addr { host: "127.6.33.60" port: 42953 } attrs { promote: true } } }
I20260926 16:31:29.846434 13029 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9d1dfe3b814a4549aebd349a75850ef4 (127.6.33.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7a7973add6e4409dabfab1a4d9e6b175" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 33939 } } peers { permanent_uuid: "9d1dfe3b814a4549aebd349a75850ef4" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 37487 } attrs { promote: false } } peers { permanent_uuid: "8b9d3065bcd44983a97ac74155813cdc" member_type: NON_VOTER last_known_addr { host: "127.6.33.60" port: 42953 } attrs { promote: true } } }
I20260926 16:31:29.850099 12977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:29.849612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:42953 (local address 127.0.0.1:46202)
0926 16:31:29.849790 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:29.849909 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.60:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260926 16:31:29.850332 12406 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7a7973add6e4409dabfab1a4d9e6b175 -> Peer 8b9d3065bcd44983a97ac74155813cdc (127.6.33.60:42953): Couldn't send request to peer 8b9d3065bcd44983a97ac74155813cdc. Status: Network error: Client connection negotiation failed: client connection to 127.6.33.60:42953: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 16:31:29.850500 12984 master.cc:552] Master@127.6.33.60:42953 shutdown complete.
I20260926 16:31:29.850570 12984 master_runner.cc:305] Clearing existing system tablet
I20260926 16:31:29.850950 12984 fs_manager.cc:359] Metadata directory not provided
I20260926 16:31:29.851058 12984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:31:29.850911 13166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a7973add6e4409dabfab1a4d9e6b175 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7a7973add6e4409dabfab1a4d9e6b175" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7a7973add6e4409dabfab1a4d9e6b175" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 33939 } } peers { permanent_uuid: "9d1dfe3b814a4549aebd349a75850ef4" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 37487 } attrs { promote: false } } peers { permanent_uuid: "8b9d3065bcd44983a97ac74155813cdc" member_type: NON_VOTER last_known_addr { host: "127.6.33.60" port: 42953 } attrs { promote: true } } }
I20260926 16:31:29.851214 13166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a7973add6e4409dabfab1a4d9e6b175 [sys.catalog]: This master's current role is: LEADER
I20260926 16:31:29.851470 13166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a7973add6e4409dabfab1a4d9e6b175 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7a7973add6e4409dabfab1a4d9e6b175" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7a7973add6e4409dabfab1a4d9e6b175" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 33939 } } peers { permanent_uuid: "9d1dfe3b814a4549aebd349a75850ef4" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 37487 } attrs { promote: false } } peers { permanent_uuid: "8b9d3065bcd44983a97ac74155813cdc" member_type: NON_VOTER last_known_addr { host: "127.6.33.60" port: 42953 } attrs { promote: true } } }
I20260926 16:31:29.851667 13166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a7973add6e4409dabfab1a4d9e6b175 [sys.catalog]: This master's current role is: LEADER
I20260926 16:31:29.853071 13058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7a7973add6e4409dabfab1a4d9e6b175" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7a7973add6e4409dabfab1a4d9e6b175" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 33939 } } peers { permanent_uuid: "9d1dfe3b814a4549aebd349a75850ef4" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 37487 } attrs { promote: false } } peers { permanent_uuid: "8b9d3065bcd44983a97ac74155813cdc" member_type: NON_VOTER last_known_addr { host: "127.6.33.60" port: 42953 } attrs { promote: true } } }
I20260926 16:31:29.853317 13058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:31:29.857429 12984 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.000s sys 0.007s
I20260926 16:31:29.859149 13174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:29.859604 12984 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:31:29.859845 12984 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260926 16:31:29.859948 12984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/master-2/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/master-2/wal
uuid: "8b9d3065bcd44983a97ac74155813cdc"
format_stamp: "Formatted at 2026-09-26 16:31:29 on dist-test-slave-khd3"
server_key: "1ac152f8d2cf6aad488d2eb1cb3e9dd5"
server_key_iv: "657d38907c7309f67318a1ea117539b3"
server_key_version: "encryptionkey@0"
I20260926 16:31:29.865217 12984 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 8b9d3065bcd44983a97ac74155813cdc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:31:29.872494 12984 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 8b9d3065bcd44983a97ac74155813cdc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 16:31:29.872676 12984 log.cc:1196] T 00000000000000000000000000000000 P 8b9d3065bcd44983a97ac74155813cdc: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260926 16:31:29.873165 12984 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 8b9d3065bcd44983a97ac74155813cdc: Deleting consensus metadata
I20260926 16:31:29.873420 12984 master_runner.cc:315] Copying system tablet from 127.6.33.62:33939
I20260926 16:31:29.876231 12984 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 8b9d3065bcd44983a97ac74155813cdc: tablet copy: Beginning tablet copy session from remote peer at address 127.6.33.62:33939
I20260926 16:31:29.896003 12747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:29.877161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50108 (local address 127.6.33.62:33939)
0926 16:31:29.877392 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:29.877398 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:29.881398 (+ 4000us) server_negotiation.cc:407] Connection header received
0926 16:31:29.881583 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:29.881589 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:29.881680 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:29.881805 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:29.882825 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:29.883959 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:29.885897 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:29.886577 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:29.886766 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:29.889243 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:29.889250 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:29.889256 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:29.889297 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:29.893043 (+ 3746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:29.893846 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:29.893852 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:29.893856 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:29.894018 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:29.894455 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:29.894460 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:29.894464 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:29.894807 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:29.894968 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:29.895420 (+ 452us) server_negotiation.cc:299] Negotiation successful
0926 16:31:29.895646 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260926 16:31:29.896066 13180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:29.877157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:33939 (local address 127.0.0.1:50108)
0926 16:31:29.881243 (+ 4086us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:29.881269 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:29.881446 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:29.881842 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:29.881851 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:29.882361 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:29.882639 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:29.882650 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:29.884139 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:29.884145 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:29.885731 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:29.885746 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:29.886779 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:29.886793 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:29.886971 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:29.887750 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:29.887776 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:29.888881 (+ 1105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:31:29.893193 (+ 4312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:29.893198 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:29.893203 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:29.893694 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:29.894140 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:29.894145 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:29.894149 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:29.894321 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:29.894936 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:31:29.894941 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:31:29.895144 (+ 203us) client_negotiation.cc:769] Sending connection context
0926 16:31:29.895286 (+ 142us) client_negotiation.cc:240] Negotiation successful
0926 16:31:29.895629 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3905,"thread_start_us":3770,"threads_started":1}
I20260926 16:31:29.896977 12442 tablet_copy_service.cc:140] P 7a7973add6e4409dabfab1a4d9e6b175: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 8b9d3065bcd44983a97ac74155813cdc ({username='oryx', principal='oryx/127.6.33.60@KRBTEST.COM'} at 127.0.0.1:50108)
I20260926 16:31:29.897145 12442 tablet_copy_service.cc:161] P 7a7973add6e4409dabfab1a4d9e6b175: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 8b9d3065bcd44983a97ac74155813cdc at {username='oryx', principal='oryx/127.6.33.60@KRBTEST.COM'} at 127.0.0.1:50108: session id = 8b9d3065bcd44983a97ac74155813cdc-00000000000000000000000000000000
I20260926 16:31:29.900691 12442 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 7a7973add6e4409dabfab1a4d9e6b175: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 16:31:29.902730 12984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 16:31:29.909444 12984 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 8b9d3065bcd44983a97ac74155813cdc: tablet copy: Starting download of 0 data blocks...
I20260926 16:31:29.909731 12984 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 8b9d3065bcd44983a97ac74155813cdc: tablet copy: Starting download of 1 WAL segments...
I20260926 16:31:29.912261 12984 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 8b9d3065bcd44983a97ac74155813cdc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 16:31:29.916558 12442 tablet_copy_service.cc:342] P 7a7973add6e4409dabfab1a4d9e6b175: Request end of tablet copy session 8b9d3065bcd44983a97ac74155813cdc-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.6.33.60@KRBTEST.COM'} at 127.0.0.1:50108
I20260926 16:31:29.916697 12442 tablet_copy_service.cc:433] P 7a7973add6e4409dabfab1a4d9e6b175: ending tablet copy session 8b9d3065bcd44983a97ac74155813cdc-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 8b9d3065bcd44983a97ac74155813cdc
I20260926 16:31:29.922348 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 12483
I20260926 16:31:29.936811 12984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:31:29.939535 13181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:31:29.945736 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.1:38529
--local_ip_for_outbound_sockets=127.6.33.1
--tserver_master_addrs=127.6.33.62:33939
--webserver_port=36175
--webserver_interface=127.6.33.1
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/krb5kdc/oryx_127.6.33.1.keytab
--principal=oryx/127.6.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:42049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.6.33.62:33939,127.6.33.61:37487
--tserver_master_addrs=127.6.33.62:33939,127.6.33.61:37487,127.6.33.60:42953 with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:31:29.945864 13182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:29.953837 13184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:31:29.954131 12984 server_base.cc:1034] running on GCE node
I20260926 16:31:29.954465 12984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:31:29.955139 12984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:31:29.956328 12984 hybrid_clock.cc:648] HybridClock initialized: now 1790440289956284 us; error 36 us; skew 500 ppm
I20260926 16:31:29.956570 12984 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:31:29 dist-test-slave-khd3 krb5kdc[12377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440289, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.33.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:31:29.958743 12984 init.cc:377] Logged in from keytab as oryx/127.6.33.60@KRBTEST.COM (short username oryx)
I20260926 16:31:29.961763 12984 webserver.cc:460] Webserver started at http://127.6.33.60:33647/ using document root <none> and password file <none>
I20260926 16:31:29.962105 12984 fs_manager.cc:359] Metadata directory not provided
I20260926 16:31:29.962203 12984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:31:29.965376 12984 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260926 16:31:29.967724 13192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:29.968222 12984 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 16:31:29.968396 12984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/master-2/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/master-2/wal
uuid: "8b9d3065bcd44983a97ac74155813cdc"
format_stamp: "Formatted at 2026-09-26 16:31:29 on dist-test-slave-khd3"
server_key: "1ac152f8d2cf6aad488d2eb1cb3e9dd5"
server_key_iv: "657d38907c7309f67318a1ea117539b3"
server_key_version: "encryptionkey@0"
I20260926 16:31:29.968657 12984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:31:29.986020 12984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:31:29.990244 12984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:31:30.017750 12984 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.60:42953
I20260926 16:31:30.017771 13243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.60:42953 every 8 connection(s)
I20260926 16:31:30.018900 12984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/master-2/data/info.pb
I20260926 16:31:30.024333 13244 sys_catalog.cc:263] Verifying existing consensus state
I20260926 16:31:30.028581 13244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b9d3065bcd44983a97ac74155813cdc: Bootstrap starting.
I20260926 16:31:30.053444 13244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b9d3065bcd44983a97ac74155813cdc: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 16:31:30.054019 13244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b9d3065bcd44983a97ac74155813cdc: Bootstrap complete.
I20260926 16:31:30.055215 13244 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b9d3065bcd44983a97ac74155813cdc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7a7973add6e4409dabfab1a4d9e6b175" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 33939 } } peers { permanent_uuid: "9d1dfe3b814a4549aebd349a75850ef4" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 37487 } attrs { promote: false } } peers { permanent_uuid: "8b9d3065bcd44983a97ac74155813cdc" member_type: NON_VOTER last_known_addr { host: "127.6.33.60" port: 42953 } attrs { promote: true } }
I20260926 16:31:30.055506 13244 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b9d3065bcd44983a97ac74155813cdc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8b9d3065bcd44983a97ac74155813cdc, State: Initialized, Role: LEARNER
I20260926 16:31:30.055756 13244 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b9d3065bcd44983a97ac74155813cdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7a7973add6e4409dabfab1a4d9e6b175" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 33939 } } peers { permanent_uuid: "9d1dfe3b814a4549aebd349a75850ef4" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 37487 } attrs { promote: false } } peers { permanent_uuid: "8b9d3065bcd44983a97ac74155813cdc" member_type: NON_VOTER last_known_addr { host: "127.6.33.60" port: 42953 } attrs { promote: true } }
I20260926 16:31:30.056663 13247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b9d3065bcd44983a97ac74155813cdc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7a7973add6e4409dabfab1a4d9e6b175" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 33939 } } peers { permanent_uuid: "9d1dfe3b814a4549aebd349a75850ef4" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 37487 } attrs { promote: false } } peers { permanent_uuid: "8b9d3065bcd44983a97ac74155813cdc" member_type: NON_VOTER last_known_addr { host: "127.6.33.60" port: 42953 } attrs { promote: true } } }
I20260926 16:31:30.057037 13244 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b9d3065bcd44983a97ac74155813cdc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:31:30.057169 13247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b9d3065bcd44983a97ac74155813cdc [sys.catalog]: This master's current role is: LEARNER
I20260926 16:31:30.066283 13258 catalog_manager.cc:1259] Loaded cluster ID: cb13d77c4ee3430abf37211ff83e1ee5
I20260926 16:31:30.066406 13258 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8b9d3065bcd44983a97ac74155813cdc: loading cluster ID for follower catalog manager: success
I20260926 16:31:30.070115 13258 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8b9d3065bcd44983a97ac74155813cdc: acquiring CA information for follower catalog manager: success
I20260926 16:31:30.072644 13258 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8b9d3065bcd44983a97ac74155813cdc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 16:31:30.330524 13186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:31:30.330856 13186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:31:30.331035 13186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:31:30.338724 13186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:31:30.338892 13186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:31:30.339071 13186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.1
I20260926 16:31:30.349223 13186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:42049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/krb5kdc/oryx_127.6.33.1.keytab
--principal=oryx/127.6.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.1:38529
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.33.1
--webserver_port=36175
--tserver_master_addrs=127.6.33.62:33939,127.6.33.61:37487,127.6.33.60:42953
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.1
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:31:30.350719 13186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:31:30.352072 13186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:31:30.366113 13263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:30.366127 13264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:30.366113 13266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:31:30.367564 13186 server_base.cc:1034] running on GCE node
I20260926 16:31:30.368243 13186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:31:30.369448 13186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:31:30.370700 13186 hybrid_clock.cc:648] HybridClock initialized: now 1790440290370631 us; error 67 us; skew 500 ppm
I20260926 16:31:30.371074 13186 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:31:30 dist-test-slave-khd3 krb5kdc[12377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440290, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:31:30.374676 13186 init.cc:377] Logged in from keytab as oryx/127.6.33.1@KRBTEST.COM (short username oryx)
I20260926 16:31:30.376227 13186 webserver.cc:460] Webserver started at http://127.6.33.1:36175/ using document root <none> and password file <none>
I20260926 16:31:30.377061 13186 fs_manager.cc:359] Metadata directory not provided
I20260926 16:31:30.377219 13186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:31:30.383219 13186 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260926 16:31:30.386178 13273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:30.388048 13186 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:31:30.388233 13186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-0/wal
uuid: "b1109a56871a442584d8a397a4c72542"
format_stamp: "Formatted at 2026-09-26 16:31:27 on dist-test-slave-khd3"
server_key: "6f8774f1fab95e3d5bf05d142b51e3cd"
server_key_iv: "577613021127d607389ee5aeb2fe8fc1"
server_key_version: "encryptionkey@0"
I20260926 16:31:30.389858 13186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:31:30.425366 13186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:31:30.430461 13186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:31:30.430706 13186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:31:30.431726 13186 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:31:30.433251 13186 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:31:30.433341 13186 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:30.433423 13186 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:31:30.433485 13186 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:30.446439 13290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:30.438517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:42953 (local address 127.0.0.1:46218)
0926 16:31:30.439485 (+ 968us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:30.439516 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:30.439709 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:30.440191 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:30.440196 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:30.440224 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:31:30.440419 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:30.440429 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:30.442403 (+ 1974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:30.442412 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:30.443893 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:30.443915 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:30.445848 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:30.445853 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:30.446002 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:30.446028 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 16:31:30.446101 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 16:31:30.446200 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":763,"thread_start_us":659,"threads_started":1}
I20260926 16:31:30.446843 13291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:30.438561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46218 (local address 127.6.33.60:42953)
0926 16:31:30.439807 (+ 1246us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:30.439814 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:30.439840 (+ 26us) server_negotiation.cc:407] Connection header received
0926 16:31:30.439921 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:30.439928 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:30.440026 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:30.440174 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:31:30.440583 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:30.442246 (+ 1663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:30.444103 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:30.445625 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:30.445800 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:30.446133 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:30.446202 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 16:31:30.446393 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1081,"thread_start_us":971,"threads_started":1}
I20260926 16:31:30.447775 13219 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 8b9d3065bcd44983a97ac74155813cdc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260926 16:31:30.448527 13247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b9d3065bcd44983a97ac74155813cdc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a7973add6e4409dabfab1a4d9e6b175. Latest consensus state: current_term: 1 leader_uuid: "7a7973add6e4409dabfab1a4d9e6b175" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7a7973add6e4409dabfab1a4d9e6b175" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 33939 } } peers { permanent_uuid: "9d1dfe3b814a4549aebd349a75850ef4" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 37487 } attrs { promote: false } } peers { permanent_uuid: "8b9d3065bcd44983a97ac74155813cdc" member_type: NON_VOTER last_known_addr { host: "127.6.33.60" port: 42953 } attrs { promote: true } } }
I20260926 16:31:30.448916 13247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b9d3065bcd44983a97ac74155813cdc [sys.catalog]: This master's current role is: LEARNER
I20260926 16:31:30.477756 13186 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.1:38529
I20260926 16:31:30.477850 13388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.1:38529 every 8 connection(s)
I20260926 16:31:30.479375 13186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
I20260926 16:31:30.482952 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 13186
I20260926 16:31:30.483253 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 12617
Sep 26 16:31:30 dist-test-slave-khd3 krb5kdc[12377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440290, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.33.1@KRBTEST.COM for oryx/127.6.33.60@KRBTEST.COM
I20260926 16:31:30.510071 13291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:30.483169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:38053 (local address 127.6.33.60:42953)
0926 16:31:30.483837 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:30.483846 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:30.490723 (+ 6877us) server_negotiation.cc:407] Connection header received
0926 16:31:30.494905 (+ 4182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:30.494910 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:30.495006 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:30.495122 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:30.497040 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:30.497752 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:30.499445 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:30.499892 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:30.500038 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:30.504234 (+ 4196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:30.504239 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:30.504242 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:30.504275 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:30.506898 (+ 2623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:30.507777 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:30.507783 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:30.507786 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:30.507912 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:30.508337 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:30.508341 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:30.508345 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:30.508620 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:30.508803 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:30.509502 (+ 699us) server_negotiation.cc:299] Negotiation successful
0926 16:31:30.509731 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486}
I20260926 16:31:30.511673 13394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:30.483303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:42953 (local address 127.6.33.1:38053)
0926 16:31:30.490501 (+ 7198us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:30.490577 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:30.494018 (+ 3441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:30.495401 (+ 1383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:30.495411 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:30.495985 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:30.496837 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:30.496863 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:30.497908 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:30.497914 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:30.499295 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:30.499307 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:30.500141 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:30.500149 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:30.500418 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:30.501202 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:30.501225 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:30.503992 (+ 2767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:31:30.507149 (+ 3157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:30.507161 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:30.507182 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:30.507643 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:30.508024 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:30.508030 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:30.508035 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:30.508217 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:30.508748 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:31:30.508780 (+ 32us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:31:30.509142 (+ 362us) client_negotiation.cc:769] Sending connection context
0926 16:31:30.509414 (+ 272us) client_negotiation.cc:240] Negotiation successful
0926 16:31:30.509975 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6758,"thread_start_us":6491,"threads_started":1}
I20260926 16:31:30.513468 13389 heartbeater.cc:347] Connected to a master server at 127.6.33.60:42953
I20260926 16:31:30.513808 13389 heartbeater.cc:464] Registering TS with master...
I20260926 16:31:30.514838 13389 heartbeater.cc:511] Master 127.6.33.60:42953 requested a full tablet report, sending...
I20260926 16:31:30.517356 13209 ts_manager.cc:194] Registered new tserver with Master: b1109a56871a442584d8a397a4c72542 (127.6.33.1:38529)
Sep 26 16:31:30 dist-test-slave-khd3 krb5kdc[12377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.33.2@KRBTEST.COM
Sep 26 16:31:30 dist-test-slave-khd3 krb5kdc[12377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440290, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.33.1@KRBTEST.COM for oryx/127.6.33.61@KRBTEST.COM
Sep 26 16:31:30 dist-test-slave-khd3 krb5kdc[12377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440290, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.33.1@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM
I20260926 16:31:30.533054 13394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:30.484046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.61:37487 (local address 127.6.33.1:52525)
0926 16:31:30.512596 (+ 28550us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:30.512616 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:30.512734 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:30.513318 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:30.513325 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:30.516371 (+ 3046us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:30.516579 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:30.516591 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:30.518090 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:30.518096 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:30.519733 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:30.519748 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:30.520670 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:30.520678 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:30.520946 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:30.521562 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:30.522662 (+ 1100us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:30.525968 (+ 3306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:31:30.529929 (+ 3961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:30.529934 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:30.529938 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:30.530454 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:30.530906 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:30.530910 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:30.530914 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:30.531136 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:30.531797 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:31:30.531802 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:31:30.531949 (+ 147us) client_negotiation.cc:769] Sending connection context
0926 16:31:30.532095 (+ 146us) client_negotiation.cc:240] Negotiation successful
0926 16:31:30.532382 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28391,"mutex_wait_us":2,"thread_start_us":30449,"threads_started":1}
I20260926 16:31:30.533174 13393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:30.482683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:34907 (local address 127.6.33.62:33939)
0926 16:31:30.494747 (+ 12064us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:30.494752 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:30.515135 (+ 20383us) server_negotiation.cc:407] Connection header received
0926 16:31:30.515421 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:30.515427 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:30.515519 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:30.515624 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:30.518900 (+ 3276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:30.519971 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:30.521626 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:30.522342 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:30.522483 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:30.526499 (+ 4016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:30.526507 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:30.526514 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:30.526554 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:30.529866 (+ 3312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:30.531053 (+ 1187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:30.531058 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:30.531061 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:30.531192 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:30.531611 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:30.531616 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:30.531619 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:30.531980 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:30.532174 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:30.532549 (+ 375us) server_negotiation.cc:299] Negotiation successful
0926 16:31:30.532795 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11938,"thread_start_us":11808,"threads_started":1}
I20260926 16:31:30.533614 13398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:30.484650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:52525 (local address 127.6.33.61:37487)
0926 16:31:30.508920 (+ 24270us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:30.508929 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:30.512870 (+ 3941us) server_negotiation.cc:407] Connection header received
0926 16:31:30.512940 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:30.512945 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:30.513038 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:30.513137 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:30.516801 (+ 3664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:30.517923 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:30.519868 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:30.520518 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:30.520642 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:30.526181 (+ 5539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:30.526186 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:30.526189 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:30.526223 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:30.529771 (+ 3548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:30.530587 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:30.530593 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:30.530597 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:30.530742 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:30.531271 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:30.531276 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:30.531281 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:30.531601 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:30.531737 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:30.532945 (+ 1208us) server_negotiation.cc:299] Negotiation successful
0926 16:31:30.533227 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24065,"thread_start_us":25475,"threads_started":1}
I20260926 16:31:30.534276 13399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:30.485157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40164 (local address 127.6.33.2:36957)
0926 16:31:30.509467 (+ 24310us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:30.509486 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:30.509524 (+ 38us) server_negotiation.cc:407] Connection header received
0926 16:31:30.509609 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:30.509621 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:30.509949 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:30.512985 (+ 3036us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:30.513330 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:30.515877 (+ 2547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:30.517680 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:30.518665 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:30.518813 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:30.526026 (+ 7213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:30.526046 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:30.526067 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:30.526113 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:30.529481 (+ 3368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:30.530366 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:30.530373 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:30.530376 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:30.530611 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:30.531073 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:30.531077 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:30.531079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:30.531504 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:30.531745 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:30.533712 (+ 1967us) server_negotiation.cc:299] Negotiation successful
0926 16:31:30.533949 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24079,"thread_start_us":30850,"threads_started":1}
I20260926 16:31:30.534693 13391 heartbeater.cc:347] Connected to a master server at 127.6.33.61:37487
I20260926 16:31:30.534844 13391 heartbeater.cc:464] Registering TS with master...
I20260926 16:31:30.535246 13391 heartbeater.cc:511] Master 127.6.33.61:37487 requested a full tablet report, sending...
I20260926 16:31:30.535547 13396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:30.484505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:33939 (local address 127.6.33.1:34907)
0926 16:31:30.515014 (+ 30509us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:30.515041 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:30.515233 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:30.515666 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:30.515670 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:30.518354 (+ 2684us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:30.518747 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:30.518762 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:30.520094 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:30.520099 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:30.521510 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:30.521522 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:30.522531 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:30.522535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:30.522648 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:30.523195 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:30.523211 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:30.526267 (+ 3056us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:31:30.530367 (+ 4100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:30.530373 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:30.530378 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:30.530922 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:30.531310 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:30.531315 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:30.531318 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:30.531480 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:30.532103 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:31:30.532108 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:31:30.532277 (+ 169us) client_negotiation.cc:769] Sending connection context
0926 16:31:30.532418 (+ 141us) client_negotiation.cc:240] Negotiation successful
0926 16:31:30.532712 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30278,"mutex_wait_us":37,"thread_start_us":30418,"threads_started":1}
I20260926 16:31:30.535759 12740 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:31:30.536450 13390 heartbeater.cc:347] Connected to a master server at 127.6.33.62:33939
I20260926 16:31:30.536612 13390 heartbeater.cc:464] Registering TS with master...
I20260926 16:31:30.537191 13390 heartbeater.cc:511] Master 127.6.33.62:33939 requested a full tablet report, sending...
I20260926 16:31:30.537232 13020 ts_manager.cc:194] Registered new tserver with Master: b1109a56871a442584d8a397a4c72542 (127.6.33.1:38529)
I20260926 16:31:30.538595 12422 ts_manager.cc:194] Re-registered known tserver with Master: b1109a56871a442584d8a397a4c72542 (127.6.33.1:38529)
I20260926 16:31:30.539438 12422 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.33.1@KRBTEST.COM'} at 127.6.33.1:34907
I20260926 16:31:30.620262 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 12617
I20260926 16:31:30.636607 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.2:36957
--local_ip_for_outbound_sockets=127.6.33.2
--tserver_master_addrs=127.6.33.62:33939
--webserver_port=38903
--webserver_interface=127.6.33.2
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/krb5kdc/oryx_127.6.33.2.keytab
--principal=oryx/127.6.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:42049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.6.33.62:33939,127.6.33.61:37487
--tserver_master_addrs=127.6.33.62:33939,127.6.33.61:37487,127.6.33.60:42953 with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:31:30.843209 13285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a7973add6e4409dabfab1a4d9e6b175 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7a7973add6e4409dabfab1a4d9e6b175" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7a7973add6e4409dabfab1a4d9e6b175" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 33939 } } peers { permanent_uuid: "9d1dfe3b814a4549aebd349a75850ef4" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 37487 } attrs { promote: false } } peers { permanent_uuid: "8b9d3065bcd44983a97ac74155813cdc" member_type: NON_VOTER last_known_addr { host: "127.6.33.60" port: 42953 } attrs { promote: true } } }
I20260926 16:31:30.843780 13285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a7973add6e4409dabfab1a4d9e6b175 [sys.catalog]: This master's current role is: LEADER
I20260926 16:31:30.845556 13402 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 7a7973add6e4409dabfab1a4d9e6b175: attempting to promote NON_VOTER 8b9d3065bcd44983a97ac74155813cdc to VOTER
I20260926 16:31:30.846172 13402 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a7973add6e4409dabfab1a4d9e6b175 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7a7973add6e4409dabfab1a4d9e6b175" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 33939 } } peers { permanent_uuid: "9d1dfe3b814a4549aebd349a75850ef4" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 37487 } attrs { promote: false } } peers { permanent_uuid: "8b9d3065bcd44983a97ac74155813cdc" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 42953 } attrs { promote: false } }
I20260926 16:31:30.848063 13029 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4 [term 1 FOLLOWER]: Refusing update from remote peer 7a7973add6e4409dabfab1a4d9e6b175: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260926 16:31:30.848095 13219 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8b9d3065bcd44983a97ac74155813cdc [term 1 LEARNER]: Refusing update from remote peer 7a7973add6e4409dabfab1a4d9e6b175: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260926 16:31:30.848702 13403 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7a7973add6e4409dabfab1a4d9e6b175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b9d3065bcd44983a97ac74155813cdc" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 42953 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260926 16:31:30.849375 13285 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7a7973add6e4409dabfab1a4d9e6b175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d1dfe3b814a4549aebd349a75850ef4" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 37487 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260926 16:31:30.852638 13403 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7a7973add6e4409dabfab1a4d9e6b175 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8b9d3065bcd44983a97ac74155813cdc (127.6.33.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7a7973add6e4409dabfab1a4d9e6b175" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 33939 } } peers { permanent_uuid: "9d1dfe3b814a4549aebd349a75850ef4" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 37487 } attrs { promote: false } } peers { permanent_uuid: "8b9d3065bcd44983a97ac74155813cdc" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 42953 } attrs { promote: false } } }
I20260926 16:31:30.853309 13029 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8b9d3065bcd44983a97ac74155813cdc (127.6.33.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7a7973add6e4409dabfab1a4d9e6b175" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 33939 } } peers { permanent_uuid: "9d1dfe3b814a4549aebd349a75850ef4" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 37487 } attrs { promote: false } } peers { permanent_uuid: "8b9d3065bcd44983a97ac74155813cdc" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 42953 } attrs { promote: false } } }
I20260926 16:31:30.854185 13219 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8b9d3065bcd44983a97ac74155813cdc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8b9d3065bcd44983a97ac74155813cdc (127.6.33.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7a7973add6e4409dabfab1a4d9e6b175" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 33939 } } peers { permanent_uuid: "9d1dfe3b814a4549aebd349a75850ef4" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 37487 } attrs { promote: false } } peers { permanent_uuid: "8b9d3065bcd44983a97ac74155813cdc" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 42953 } attrs { promote: false } } }
I20260926 16:31:30.857491 13285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a7973add6e4409dabfab1a4d9e6b175 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7a7973add6e4409dabfab1a4d9e6b175" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7a7973add6e4409dabfab1a4d9e6b175" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 33939 } } peers { permanent_uuid: "9d1dfe3b814a4549aebd349a75850ef4" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 37487 } attrs { promote: false } } peers { permanent_uuid: "8b9d3065bcd44983a97ac74155813cdc" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 42953 } attrs { promote: false } } }
I20260926 16:31:30.857725 13285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a7973add6e4409dabfab1a4d9e6b175 [sys.catalog]: This master's current role is: LEADER
I20260926 16:31:30.858011 13285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a7973add6e4409dabfab1a4d9e6b175 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7a7973add6e4409dabfab1a4d9e6b175" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7a7973add6e4409dabfab1a4d9e6b175" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 33939 } } peers { permanent_uuid: "9d1dfe3b814a4549aebd349a75850ef4" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 37487 } attrs { promote: false } } peers { permanent_uuid: "8b9d3065bcd44983a97ac74155813cdc" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 42953 } attrs { promote: false } } }
I20260926 16:31:30.858222 13285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a7973add6e4409dabfab1a4d9e6b175 [sys.catalog]: This master's current role is: LEADER
I20260926 16:31:30.858584 13247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b9d3065bcd44983a97ac74155813cdc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7a7973add6e4409dabfab1a4d9e6b175" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7a7973add6e4409dabfab1a4d9e6b175" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 33939 } } peers { permanent_uuid: "9d1dfe3b814a4549aebd349a75850ef4" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 37487 } attrs { promote: false } } peers { permanent_uuid: "8b9d3065bcd44983a97ac74155813cdc" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 42953 } attrs { promote: false } } }
I20260926 16:31:30.858620 13285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a7973add6e4409dabfab1a4d9e6b175 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7a7973add6e4409dabfab1a4d9e6b175" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7a7973add6e4409dabfab1a4d9e6b175" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 33939 } } peers { permanent_uuid: "9d1dfe3b814a4549aebd349a75850ef4" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 37487 } attrs { promote: false } } peers { permanent_uuid: "8b9d3065bcd44983a97ac74155813cdc" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 42953 } attrs { promote: false } } }
I20260926 16:31:30.858843 13247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b9d3065bcd44983a97ac74155813cdc [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:31:30.858901 13285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a7973add6e4409dabfab1a4d9e6b175 [sys.catalog]: This master's current role is: LEADER
I20260926 16:31:30.859694 13407 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7a7973add6e4409dabfab1a4d9e6b175" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7a7973add6e4409dabfab1a4d9e6b175" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 33939 } } peers { permanent_uuid: "9d1dfe3b814a4549aebd349a75850ef4" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 37487 } attrs { promote: false } } peers { permanent_uuid: "8b9d3065bcd44983a97ac74155813cdc" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 42953 } attrs { promote: false } } }
I20260926 16:31:30.860123 13407 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 16:31:30.972916 13401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:31:30.973209 13401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:31:30.973305 13401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:31:30.981097 13401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:31:30.981211 13401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:31:30.981324 13401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.2
I20260926 16:31:30.991228 13401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:42049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/krb5kdc/oryx_127.6.33.2.keytab
--principal=oryx/127.6.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.2:36957
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.33.2
--webserver_port=38903
--tserver_master_addrs=127.6.33.62:33939,127.6.33.61:37487,127.6.33.60:42953
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.2
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:31:30.992502 13401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:31:30.993851 13401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:31:31.007807 13412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:31.007850 13413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:31.007813 13415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:31:31.008747 13401 server_base.cc:1034] running on GCE node
I20260926 16:31:31.009413 13401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:31:31.010419 13401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:31:31.011646 13401 hybrid_clock.cc:648] HybridClock initialized: now 1790440291011565 us; error 78 us; skew 500 ppm
I20260926 16:31:31.011984 13401 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:31:31 dist-test-slave-khd3 krb5kdc[12377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440291, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:31:31.015658 13401 init.cc:377] Logged in from keytab as oryx/127.6.33.2@KRBTEST.COM (short username oryx)
I20260926 16:31:31.020165 13401 webserver.cc:460] Webserver started at http://127.6.33.2:38903/ using document root <none> and password file <none>
I20260926 16:31:31.021003 13401 fs_manager.cc:359] Metadata directory not provided
I20260926 16:31:31.021152 13401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:31:31.026683 13401 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 16:31:31.029798 13422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:31.031884 13401 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 16:31:31.032061 13401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-1/wal
uuid: "d295b421f67a4d0e962b6c2365248448"
format_stamp: "Formatted at 2026-09-26 16:31:27 on dist-test-slave-khd3"
server_key: "b5e2bc1969ba42b61127c3789423627d"
server_key_iv: "0d7a30e9a7f3e639a13f6ef956538be3"
server_key_version: "encryptionkey@0"
I20260926 16:31:31.033384 13401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:31:31.063077 13401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:31:31.067931 13401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:31:31.068228 13401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:31:31.069546 13401 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:31:31.071027 13401 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:31:31.071147 13401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:31.071282 13401 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:31:31.071367 13401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:31.103726 13401 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.2:36957
I20260926 16:31:31.103770 13534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.2:36957 every 8 connection(s)
I20260926 16:31:31.105209 13401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
I20260926 16:31:31.106774 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 13401
I20260926 16:31:31.107046 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 12751
Sep 26 16:31:31 dist-test-slave-khd3 krb5kdc[12377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.33.3@KRBTEST.COM
Sep 26 16:31:31 dist-test-slave-khd3 krb5kdc[12377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440291, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.33.2@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM
Sep 26 16:31:31 dist-test-slave-khd3 krb5kdc[12377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440291, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.33.2@KRBTEST.COM for oryx/127.6.33.61@KRBTEST.COM
Sep 26 16:31:31 dist-test-slave-khd3 krb5kdc[12377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440291, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.33.2@KRBTEST.COM for oryx/127.6.33.60@KRBTEST.COM
I20260926 16:31:31.154590 13541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:31.108641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58620 (local address 127.6.33.3:45149)
0926 16:31:31.125244 (+ 16603us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:31.125257 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:31.129987 (+ 4730us) server_negotiation.cc:407] Connection header received
0926 16:31:31.130345 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:31.130359 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:31.130683 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:31.130871 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:31.134354 (+ 3483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:31.135408 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:31.137047 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:31.138198 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:31.138314 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:31.145210 (+ 6896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:31.145224 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:31.145242 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:31.145292 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:31.150499 (+ 5207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:31.151957 (+ 1458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:31.151966 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:31.151969 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:31.152141 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:31.152531 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:31.152538 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:31.152542 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:31.152997 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:31.153274 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:31.153876 (+ 602us) server_negotiation.cc:299] Negotiation successful
0926 16:31:31.154140 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16389,"thread_start_us":16238,"threads_started":1}
I20260926 16:31:31.156827 12874 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:31:31.156999 13542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:31.108964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:56861 (local address 127.6.33.62:33939)
0926 16:31:31.124672 (+ 15708us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:31.124681 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:31.128822 (+ 4141us) server_negotiation.cc:407] Connection header received
0926 16:31:31.134195 (+ 5373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:31.134201 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:31.134285 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:31.134400 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:31.138124 (+ 3724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:31.139190 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:31.142010 (+ 2820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:31.142700 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:31.142857 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:31.147740 (+ 4883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:31.147747 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:31.147752 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:31.147797 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:31.151487 (+ 3690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:31.153083 (+ 1596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:31.153088 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:31.153092 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:31.153244 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:31.153747 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:31.153751 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:31.153754 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:31.154084 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:31.154234 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:31.156264 (+ 2030us) server_negotiation.cc:299] Negotiation successful
0926 16:31:31.156489 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15555,"thread_start_us":15445,"threads_started":1}
I20260926 16:31:31.158329 13540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:31.108544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:51973 (local address 127.6.33.61:37487)
0926 16:31:31.124109 (+ 15565us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:31.124118 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:31.129613 (+ 5495us) server_negotiation.cc:407] Connection header received
0926 16:31:31.131022 (+ 1409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:31.131030 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:31.131136 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:31.131257 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:31.134234 (+ 2977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:31.135658 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:31.140911 (+ 5253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:31.141546 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:31.141646 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:31.148499 (+ 6853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:31.148505 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:31.148511 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:31.148554 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:31.151855 (+ 3301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:31.153059 (+ 1204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:31.153066 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:31.153071 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:31.153221 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:31.154302 (+ 1081us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:31.154308 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:31.154311 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:31.154674 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:31.154839 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:31.157585 (+ 2746us) server_negotiation.cc:299] Negotiation successful
0926 16:31:31.157826 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15383,"thread_start_us":15585,"threads_started":1}
I20260926 16:31:31.158403 13543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:31.108870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.61:37487 (local address 127.6.33.2:51973)
0926 16:31:31.129392 (+ 20522us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:31.129424 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:31.130511 (+ 1087us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:31.131793 (+ 1282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:31.131809 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:31.132930 (+ 1121us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:31.133895 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:31.133928 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:31.135887 (+ 1959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:31.135895 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:31.139961 (+ 4066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:31.139979 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:31.143059 (+ 3080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:31.143071 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:31.143531 (+ 460us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:31.144321 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:31.144352 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:31.148272 (+ 3920us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:31:31.151986 (+ 3714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:31.151994 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:31.152012 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:31.152920 (+ 908us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:31.153392 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:31.153398 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:31.153401 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:31.153620 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:31.154900 (+ 1280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:31:31.154906 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:31:31.155318 (+ 412us) client_negotiation.cc:769] Sending connection context
0926 16:31:31.155678 (+ 360us) client_negotiation.cc:240] Negotiation successful
0926 16:31:31.156949 (+ 1271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20321,"mutex_wait_us":765,"spinlock_wait_cycles":15104,"thread_start_us":16495,"threads_started":1}
I20260926 16:31:31.159313 13545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:31.126156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:50387 (local address 127.6.33.60:42953)
0926 16:31:31.134243 (+ 8087us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:31.134249 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:31.140926 (+ 6677us) server_negotiation.cc:407] Connection header received
0926 16:31:31.141197 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:31.141203 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:31.141304 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:31.141388 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:31.144022 (+ 2634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:31.145236 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:31.146879 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:31.147513 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:31.147628 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:31.152277 (+ 4649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:31.152286 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:31.152292 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:31.152336 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:31.155418 (+ 3082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:31.156162 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:31.156167 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:31.156169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:31.156318 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:31.156742 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:31.156747 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:31.156750 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:31.157285 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:31.157512 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:31.158305 (+ 793us) server_negotiation.cc:299] Negotiation successful
0926 16:31:31.158735 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7965,"thread_start_us":7840,"threads_started":1}
I20260926 16:31:31.160151 13546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:31.126177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:42953 (local address 127.6.33.2:50387)
0926 16:31:31.140789 (+ 14612us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:31.140814 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:31.141035 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:31.141637 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:31.141642 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:31.143256 (+ 1614us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:31.143859 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:31.143873 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:31.145397 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:31.145401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:31.146716 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:31.146730 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:31.147940 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:31.147948 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:31.148141 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:31.148702 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:31.148727 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:31.151885 (+ 3158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:31:31.155548 (+ 3663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:31.155552 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:31.155555 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:31.156022 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:31.156446 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:31.156452 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:31.156455 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:31.156614 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:31.157586 (+ 972us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:31:31.157594 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:31:31.157771 (+ 177us) client_negotiation.cc:769] Sending connection context
0926 16:31:31.158011 (+ 240us) client_negotiation.cc:240] Negotiation successful
0926 16:31:31.159869 (+ 1858us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14439,"mutex_wait_us":71,"thread_start_us":14239,"threads_started":1}
I20260926 16:31:31.160528 13537 heartbeater.cc:347] Connected to a master server at 127.6.33.61:37487
I20260926 16:31:31.160917 13537 heartbeater.cc:464] Registering TS with master...
I20260926 16:31:31.161024 13535 heartbeater.cc:347] Connected to a master server at 127.6.33.60:42953
I20260926 16:31:31.161164 13535 heartbeater.cc:464] Registering TS with master...
I20260926 16:31:31.161855 13544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:31.108869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:33939 (local address 127.6.33.2:56861)
0926 16:31:31.125982 (+ 17113us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:31.126045 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:31.134068 (+ 8023us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:31.136866 (+ 2798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:31.136874 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:31.137496 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:31.137827 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:31.137841 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:31.140145 (+ 2304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:31.140153 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:31.141883 (+ 1730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:31.141896 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:31.143339 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:31.143345 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:31.143615 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:31.144383 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:31.144411 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:31.147492 (+ 3081us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:31:31.152035 (+ 4543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:31.152040 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:31.152043 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:31.152786 (+ 743us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:31.153371 (+ 585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:31.153375 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:31.153378 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:31.153626 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:31.154249 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:31:31.154259 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:31:31.155718 (+ 1459us) client_negotiation.cc:769] Sending connection context
0926 16:31:31.155848 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 16:31:31.156309 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16777,"mutex_wait_us":130,"spinlock_wait_cycles":25472,"thread_start_us":20045,"threads_started":1}
I20260926 16:31:31.162283 13535 heartbeater.cc:511] Master 127.6.33.60:42953 requested a full tablet report, sending...
I20260926 16:31:31.162493 13537 heartbeater.cc:511] Master 127.6.33.61:37487 requested a full tablet report, sending...
I20260926 16:31:31.163319 13536 heartbeater.cc:347] Connected to a master server at 127.6.33.62:33939
I20260926 16:31:31.163450 13536 heartbeater.cc:464] Registering TS with master...
I20260926 16:31:31.163872 13536 heartbeater.cc:511] Master 127.6.33.62:33939 requested a full tablet report, sending...
I20260926 16:31:31.164184 13209 ts_manager.cc:194] Registered new tserver with Master: d295b421f67a4d0e962b6c2365248448 (127.6.33.2:36957)
I20260926 16:31:31.164222 13020 ts_manager.cc:194] Registered new tserver with Master: d295b421f67a4d0e962b6c2365248448 (127.6.33.2:36957)
I20260926 16:31:31.165024 12422 ts_manager.cc:194] Re-registered known tserver with Master: d295b421f67a4d0e962b6c2365248448 (127.6.33.2:36957)
I20260926 16:31:31.166157 12422 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.33.2@KRBTEST.COM'} at 127.6.33.2:56861
I20260926 16:31:31.248349 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 12751
I20260926 16:31:31.267997 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.3:45149
--local_ip_for_outbound_sockets=127.6.33.3
--tserver_master_addrs=127.6.33.62:33939
--webserver_port=43881
--webserver_interface=127.6.33.3
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/krb5kdc/oryx_127.6.33.3.keytab
--principal=oryx/127.6.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:42049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.6.33.62:33939,127.6.33.61:37487
--tserver_master_addrs=127.6.33.62:33939,127.6.33.61:37487,127.6.33.60:42953 with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:31:31.542335 13390 heartbeater.cc:503] Master 127.6.33.62:33939 was elected leader, sending a full tablet report...
W20260926 16:31:31.596576 13548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:31:31.596915 13548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:31:31.597060 13548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:31:31.604452 13548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:31:31.604558 13548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:31:31.604671 13548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.3
I20260926 16:31:31.614656 13548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:42049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/krb5kdc/oryx_127.6.33.3.keytab
--principal=oryx/127.6.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.3:45149
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.33.3
--webserver_port=43881
--tserver_master_addrs=127.6.33.62:33939,127.6.33.61:37487,127.6.33.60:42953
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.3
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:31:31.616034 13548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:31:31.617499 13548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:31:31.634923 13553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:31.634960 13554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:31.634886 13556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:31:31.636029 13548 server_base.cc:1034] running on GCE node
I20260926 16:31:31.636637 13548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:31:31.637712 13548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:31:31.638918 13548 hybrid_clock.cc:648] HybridClock initialized: now 1790440291638867 us; error 44 us; skew 500 ppm
I20260926 16:31:31.639246 13548 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:31:31 dist-test-slave-khd3 krb5kdc[12377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440291, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:31:31.642544 13548 init.cc:377] Logged in from keytab as oryx/127.6.33.3@KRBTEST.COM (short username oryx)
I20260926 16:31:31.644006 13548 webserver.cc:460] Webserver started at http://127.6.33.3:43881/ using document root <none> and password file <none>
I20260926 16:31:31.644703 13548 fs_manager.cc:359] Metadata directory not provided
I20260926 16:31:31.644878 13548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:31:31.650107 13548 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260926 16:31:31.652711 13563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:31.654071 13548 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 16:31:31.654235 13548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-2/wal
uuid: "93c3a12e5462468b875b5d0fbdd048de"
format_stamp: "Formatted at 2026-09-26 16:31:28 on dist-test-slave-khd3"
server_key: "a4baac71b7683e819f5630ea1e36527b"
server_key_iv: "b1be76d4259a98d630ce21a25a2fe934"
server_key_version: "encryptionkey@0"
I20260926 16:31:31.655186 13548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:31:31.674264 13548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:31:31.678728 13548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:31:31.679024 13548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:31:31.680056 13548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:31:31.681632 13548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:31:31.681751 13548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:31.681874 13548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:31:31.681963 13548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:31.715775 13548 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.3:45149
I20260926 16:31:31.715824 13675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.3:45149 every 8 connection(s)
I20260926 16:31:31.717347 13548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
I20260926 16:31:31.718393 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 13548
Sep 26 16:31:31 dist-test-slave-khd3 krb5kdc[12377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440291, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.33.3@KRBTEST.COM for oryx/127.6.33.60@KRBTEST.COM
Sep 26 16:31:31 dist-test-slave-khd3 krb5kdc[12377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440291, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.33.3@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM
I20260926 16:31:31.772923 13683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:31.722593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.3:58445 (local address 127.6.33.60:42953)
0926 16:31:31.747716 (+ 25123us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:31.747723 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:31.748446 (+ 723us) server_negotiation.cc:407] Connection header received
0926 16:31:31.750292 (+ 1846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:31.750300 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:31.750390 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:31.750503 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:31.753208 (+ 2705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:31.754511 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:31.757752 (+ 3241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:31.758621 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:31.758717 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:31.764046 (+ 5329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:31.764053 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:31.764059 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:31.764102 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:31.768331 (+ 4229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:31.769858 (+ 1527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:31.769863 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:31.769866 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:31.770020 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:31.770603 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:31.770610 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:31.770617 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:31.771026 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:31.771158 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:31.772245 (+ 1087us) server_negotiation.cc:299] Negotiation successful
0926 16:31:31.772489 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24984,"thread_start_us":25130,"threads_started":1}
Sep 26 16:31:31 dist-test-slave-khd3 krb5kdc[12377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440291, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.33.3@KRBTEST.COM for oryx/127.6.33.61@KRBTEST.COM
I20260926 16:31:31.775513 13682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:31.722135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:42953 (local address 127.6.33.3:58445)
0926 16:31:31.748257 (+ 26122us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:31.748280 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:31.750129 (+ 1849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:31.750552 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:31.750557 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:31.751939 (+ 1382us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:31.753008 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:31.753033 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:31.754731 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:31.754739 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:31.757553 (+ 2814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:31.757574 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:31.759150 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:31.759158 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:31.759465 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:31.760354 (+ 889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:31.760385 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:31.763775 (+ 3390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:31:31.768510 (+ 4735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:31.768519 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:31.768540 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:31.769712 (+ 1172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:31.770180 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:31.770186 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:31.770192 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:31.770462 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:31.771257 (+ 795us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:31:31.771267 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:31:31.771739 (+ 472us) client_negotiation.cc:769] Sending connection context
0926 16:31:31.773245 (+ 1506us) client_negotiation.cc:240] Negotiation successful
0926 16:31:31.773919 (+ 674us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25990,"mutex_wait_us":61,"thread_start_us":26914,"threads_started":1}
I20260926 16:31:31.777462 13676 heartbeater.cc:347] Connected to a master server at 127.6.33.60:42953
I20260926 16:31:31.777841 13676 heartbeater.cc:464] Registering TS with master...
I20260926 16:31:31.779104 13676 heartbeater.cc:511] Master 127.6.33.60:42953 requested a full tablet report, sending...
I20260926 16:31:31.780311 13681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:31.721336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:33939 (local address 127.6.33.3:49293)
0926 16:31:31.745476 (+ 24140us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:31.745550 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:31.748586 (+ 3036us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:31.751045 (+ 2459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:31.751053 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:31.751914 (+ 861us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:31.753018 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:31.753034 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:31.761067 (+ 8033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:31.761075 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:31.763017 (+ 1942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:31.763036 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:31.765212 (+ 2176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:31.765221 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:31.765494 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:31.766173 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:31.766202 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:31.769534 (+ 3332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:31:31.776298 (+ 6764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:31.776307 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:31.776327 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:31.777037 (+ 710us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:31.777701 (+ 664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:31.777706 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:31.777709 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:31.777889 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:31.779261 (+ 1372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:31:31.779266 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:31:31.779401 (+ 135us) client_negotiation.cc:769] Sending connection context
0926 16:31:31.779543 (+ 142us) client_negotiation.cc:240] Negotiation successful
0926 16:31:31.779840 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23661,"mutex_wait_us":119,"spinlock_wait_cycles":17664,"thread_start_us":28515,"threads_started":1}
I20260926 16:31:31.780368 13680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:31.720653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.3:49293 (local address 127.6.33.62:33939)
0926 16:31:31.749350 (+ 28697us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:31.749356 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:31.749381 (+ 25us) server_negotiation.cc:407] Connection header received
0926 16:31:31.750771 (+ 1390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:31.750777 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:31.750869 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:31.750993 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:31.753209 (+ 2216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:31.760902 (+ 7693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:31.764267 (+ 3365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:31.764997 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:31.765142 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:31.769924 (+ 4782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:31.769933 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:31.769941 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:31.769984 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:31.775054 (+ 5070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:31.777354 (+ 2300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:31.777363 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:31.777368 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:31.777527 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:31.778658 (+ 1131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:31.778665 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:31.778669 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:31.779064 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:31.779222 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:31.779790 (+ 568us) server_negotiation.cc:299] Negotiation successful
0926 16:31:31.780019 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28542,"thread_start_us":28726,"threads_started":1}
I20260926 16:31:31.781520 13209 ts_manager.cc:194] Registered new tserver with Master: 93c3a12e5462468b875b5d0fbdd048de (127.6.33.3:45149)
I20260926 16:31:31.781601 13677 heartbeater.cc:347] Connected to a master server at 127.6.33.62:33939
I20260926 16:31:31.781744 13677 heartbeater.cc:464] Registering TS with master...
I20260926 16:31:31.782298 13677 heartbeater.cc:511] Master 127.6.33.62:33939 requested a full tablet report, sending...
I20260926 16:31:31.783304 12422 ts_manager.cc:194] Re-registered known tserver with Master: 93c3a12e5462468b875b5d0fbdd048de (127.6.33.3:45149)
I20260926 16:31:31.784612 12422 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.33.3@KRBTEST.COM'} at 127.6.33.3:49293
I20260926 16:31:31.786245 13685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:31.725451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.61:37487 (local address 127.6.33.3:39515)
0926 16:31:31.749575 (+ 24124us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:31.749592 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:31.749721 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:31.750425 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:31.750433 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:31.752039 (+ 1606us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:31.753008 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:31.753033 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:31.762544 (+ 9511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:31.762552 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:31.764103 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:31.764115 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:31.768292 (+ 4177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:31.768298 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:31.768484 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:31.769679 (+ 1195us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:31.769700 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:31.777994 (+ 8294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:31:31.783362 (+ 5368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:31.783369 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:31.783375 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:31.783861 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:31.784335 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:31.784350 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:31.784354 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:31.784489 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:31.785183 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:31:31.785189 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:31:31.785372 (+ 183us) client_negotiation.cc:769] Sending connection context
0926 16:31:31.785510 (+ 138us) client_negotiation.cc:240] Negotiation successful
0926 16:31:31.785810 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24019,"mutex_wait_us":36,"thread_start_us":23848,"threads_started":1}
I20260926 16:31:31.787200 13684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:31.725463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.3:39515 (local address 127.6.33.61:37487)
0926 16:31:31.749153 (+ 23690us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:31.749158 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:31.749693 (+ 535us) server_negotiation.cc:407] Connection header received
0926 16:31:31.749919 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:31.749925 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:31.750020 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:31.750166 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:31.761240 (+ 11074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:31.762311 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:31.767106 (+ 4795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:31.767818 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:31.767939 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:31.778200 (+ 10261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:31.778210 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:31.778216 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:31.778281 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:31.783182 (+ 4901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:31.784017 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:31.784024 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:31.784028 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:31.784204 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:31.784618 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:31.784624 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:31.784630 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:31.784977 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:31.785123 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:31.786643 (+ 1520us) server_negotiation.cc:299] Negotiation successful
0926 16:31:31.786859 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23515,"thread_start_us":23422,"threads_started":1}
I20260926 16:31:31.788363 13678 heartbeater.cc:347] Connected to a master server at 127.6.33.61:37487
I20260926 16:31:31.788487 13678 heartbeater.cc:464] Registering TS with master...
I20260926 16:31:31.788970 13678 heartbeater.cc:511] Master 127.6.33.61:37487 requested a full tablet report, sending...
I20260926 16:31:31.790194 13020 ts_manager.cc:194] Registered new tserver with Master: 93c3a12e5462468b875b5d0fbdd048de (127.6.33.3:45149)
I20260926 16:31:31.804689 13684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:31.754330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57152 (local address 127.6.33.61:37487)
0926 16:31:31.787442 (+ 33112us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:31.787448 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:31.787614 (+ 166us) server_negotiation.cc:407] Connection header received
0926 16:31:31.787786 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:31.787791 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:31.787863 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:31.787961 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:31.790486 (+ 2525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:31.791551 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:31.792969 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:31.793642 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:31.793733 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:31.797726 (+ 3993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:31.797732 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:31.797737 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:31.797778 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:31.800833 (+ 3055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:31.801738 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:31.801743 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:31.801747 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:31.801926 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:31.802350 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:31.802355 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:31.802359 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:31.802794 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:31.803479 (+ 685us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:31.804077 (+ 598us) server_negotiation.cc:299] Negotiation successful
0926 16:31:31.804308 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32983,"thread_start_us":40612,"threads_started":1}
I20260926 16:31:31.813549 13690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:31.753641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50120 (local address 127.6.33.62:33939)
0926 16:31:31.778522 (+ 24881us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:31.778528 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:31.795875 (+ 17347us) server_negotiation.cc:407] Connection header received
0926 16:31:31.796152 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:31.796158 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:31.796241 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:31.796326 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:31.798743 (+ 2417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:31.800334 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:31.802500 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:31.802938 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:31.803087 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:31.807634 (+ 4547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:31.807640 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:31.807644 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:31.807690 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:31.810998 (+ 3308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:31.811587 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:31.811592 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:31.811596 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:31.811724 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:31.812078 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:31.812083 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:31.812086 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:31.812352 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:31.812614 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:31.812950 (+ 336us) server_negotiation.cc:299] Negotiation successful
0926 16:31:31.813150 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24755,"thread_start_us":24892,"threads_started":1}
I20260926 16:31:31.816043 13683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:31.775352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46224 (local address 127.6.33.60:42953)
0926 16:31:31.777859 (+ 2507us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:31.777866 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:31.801515 (+ 23649us) server_negotiation.cc:407] Connection header received
0926 16:31:31.801668 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:31.801673 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:31.801764 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:31.801862 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:31.803846 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:31.804961 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:31.807055 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:31.807510 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:31.807636 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:31.810724 (+ 3088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:31.810729 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:31.810732 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:31.810764 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:31.813279 (+ 2515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:31.814031 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:31.814037 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:31.814047 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:31.814163 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:31.814530 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:31.814534 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:31.814536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:31.814889 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:31.815145 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:31.815474 (+ 329us) server_negotiation.cc:299] Negotiation successful
0926 16:31:31.815678 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2341}
I20260926 16:31:31.817981 12422 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50120:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:31:31.824983 12422 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:31:31.854683 13704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:31.844539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58622 (local address 127.6.33.3:45149)
0926 16:31:31.845992 (+ 1453us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:31.846004 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:31.846047 (+ 43us) server_negotiation.cc:407] Connection header received
0926 16:31:31.846172 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:31.846183 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:31.846542 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:31.846771 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:31:31.847384 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:31.849417 (+ 2033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:31.852077 (+ 2660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:31.853628 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:31.853744 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:31.853997 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:31.854219 (+ 222us) server_negotiation.cc:299] Negotiation successful
0926 16:31:31.854385 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1237,"thread_start_us":1069,"threads_started":1}
I20260926 16:31:31.854661 13703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:31.844184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.3:45149 (local address 127.0.0.1:58622)
0926 16:31:31.844838 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:31.844865 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:31.845062 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:31.846928 (+ 1866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:31.846932 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:31.846952 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:31:31.847202 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:31.847209 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:31.849537 (+ 2328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:31.849542 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:31.851961 (+ 2419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:31.851974 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:31.853843 (+ 1869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:31.853850 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:31.854031 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:31.854067 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 16:31:31.854165 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 16:31:31.854291 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":305,"threads_started":1}
I20260926 16:31:31.858829 13706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:31.845356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40180 (local address 127.6.33.2:36957)
0926 16:31:31.845894 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:31.845904 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:31.849255 (+ 3351us) server_negotiation.cc:407] Connection header received
0926 16:31:31.850567 (+ 1312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:31.850578 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:31.850848 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:31.851035 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:31:31.852856 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:31.854047 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:31.856341 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:31.857769 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:31.857916 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:31.858118 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:31.858356 (+ 238us) server_negotiation.cc:299] Negotiation successful
0926 16:31:31.858502 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":782,"threads_started":1}
I20260926 16:31:31.858884 13705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:31.844927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.2:36957 (local address 127.0.0.1:40180)
0926 16:31:31.849126 (+ 4199us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:31.849145 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:31.849360 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:31.851287 (+ 1927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:31.851292 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:31.851338 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:31:31.851892 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:31.851905 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:31.854369 (+ 2464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:31.854373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:31.856172 (+ 1799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:31.856186 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:31.857971 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:31.857977 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:31.858142 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:31.858174 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 16:31:31.858298 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 16:31:31.858428 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4105,"mutex_wait_us":9,"spinlock_wait_cycles":9472,"thread_start_us":4003,"threads_started":1}
I20260926 16:31:31.858564 13611 tablet_service.cc:1467] Processing CreateTablet for tablet 71779fdf45ca4138bb047d75eb576f57 (DEFAULT_TABLE table=test-table [id=ffe04a9bfaa54487a41d8374855aea26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:31:31.860286 13708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:31.850030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:38529 (local address 127.0.0.1:39916)
0926 16:31:31.850560 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:31.850574 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:31.850713 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:31.851219 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:31.851223 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:31.851248 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:31:31.851760 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:31.851771 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:31.854071 (+ 2300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:31.854075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:31.857518 (+ 3443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:31.857532 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:31.859350 (+ 1818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:31.859357 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:31.859523 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:31.859554 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 16:31:31.859637 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 16:31:31.859767 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"mutex_wait_us":33,"spinlock_wait_cycles":11904,"thread_start_us":264,"threads_started":1}
I20260926 16:31:31.860973 13707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:31.849514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39916 (local address 127.6.33.1:38529)
0926 16:31:31.850060 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:31.850071 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:31.850672 (+ 601us) server_negotiation.cc:407] Connection header received
0926 16:31:31.850887 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:31.850893 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:31.851054 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:31.851240 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:31:31.851969 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:31.853537 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:31.857684 (+ 4147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:31.859150 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:31.859273 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:31.859470 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:31.860497 (+ 1027us) server_negotiation.cc:299] Negotiation successful
0926 16:31:31.860667 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":235,"threads_started":1}
I20260926 16:31:31.861004 13611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71779fdf45ca4138bb047d75eb576f57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:31:31.862552 13470 tablet_service.cc:1467] Processing CreateTablet for tablet 71779fdf45ca4138bb047d75eb576f57 (DEFAULT_TABLE table=test-table [id=ffe04a9bfaa54487a41d8374855aea26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:31:31.863947 13470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71779fdf45ca4138bb047d75eb576f57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:31:31.864384 13324 tablet_service.cc:1467] Processing CreateTablet for tablet 71779fdf45ca4138bb047d75eb576f57 (DEFAULT_TABLE table=test-table [id=ffe04a9bfaa54487a41d8374855aea26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:31:31.865729 13324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71779fdf45ca4138bb047d75eb576f57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:31:31.874651 13709 tablet_bootstrap.cc:492] T 71779fdf45ca4138bb047d75eb576f57 P 93c3a12e5462468b875b5d0fbdd048de: Bootstrap starting.
I20260926 16:31:31.877696 13709 tablet_bootstrap.cc:654] T 71779fdf45ca4138bb047d75eb576f57 P 93c3a12e5462468b875b5d0fbdd048de: Neither blocks nor log segments found. Creating new log.
I20260926 16:31:31.878873 13709 log.cc:824] T 71779fdf45ca4138bb047d75eb576f57 P 93c3a12e5462468b875b5d0fbdd048de: Log is configured to *not* fsync() on all Append() calls
I20260926 16:31:31.879148 13710 tablet_bootstrap.cc:492] T 71779fdf45ca4138bb047d75eb576f57 P d295b421f67a4d0e962b6c2365248448: Bootstrap starting.
I20260926 16:31:31.880795 13712 tablet_bootstrap.cc:492] T 71779fdf45ca4138bb047d75eb576f57 P b1109a56871a442584d8a397a4c72542: Bootstrap starting.
I20260926 16:31:31.883595 13710 tablet_bootstrap.cc:654] T 71779fdf45ca4138bb047d75eb576f57 P d295b421f67a4d0e962b6c2365248448: Neither blocks nor log segments found. Creating new log.
I20260926 16:31:31.883925 13712 tablet_bootstrap.cc:654] T 71779fdf45ca4138bb047d75eb576f57 P b1109a56871a442584d8a397a4c72542: Neither blocks nor log segments found. Creating new log.
I20260926 16:31:31.885046 13712 log.cc:824] T 71779fdf45ca4138bb047d75eb576f57 P b1109a56871a442584d8a397a4c72542: Log is configured to *not* fsync() on all Append() calls
I20260926 16:31:31.885131 13710 log.cc:824] T 71779fdf45ca4138bb047d75eb576f57 P d295b421f67a4d0e962b6c2365248448: Log is configured to *not* fsync() on all Append() calls
I20260926 16:31:31.887679 13709 tablet_bootstrap.cc:492] T 71779fdf45ca4138bb047d75eb576f57 P 93c3a12e5462468b875b5d0fbdd048de: No bootstrap required, opened a new log
I20260926 16:31:31.887969 13709 ts_tablet_manager.cc:1397] T 71779fdf45ca4138bb047d75eb576f57 P 93c3a12e5462468b875b5d0fbdd048de: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.007s
I20260926 16:31:31.888080 13712 tablet_bootstrap.cc:492] T 71779fdf45ca4138bb047d75eb576f57 P b1109a56871a442584d8a397a4c72542: No bootstrap required, opened a new log
I20260926 16:31:31.888335 13712 ts_tablet_manager.cc:1397] T 71779fdf45ca4138bb047d75eb576f57 P b1109a56871a442584d8a397a4c72542: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260926 16:31:31.888821 13710 tablet_bootstrap.cc:492] T 71779fdf45ca4138bb047d75eb576f57 P d295b421f67a4d0e962b6c2365248448: No bootstrap required, opened a new log
I20260926 16:31:31.889082 13710 ts_tablet_manager.cc:1397] T 71779fdf45ca4138bb047d75eb576f57 P d295b421f67a4d0e962b6c2365248448: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260926 16:31:31.895793 13709 raft_consensus.cc:348] T 71779fdf45ca4138bb047d75eb576f57 P 93c3a12e5462468b875b5d0fbdd048de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c3a12e5462468b875b5d0fbdd048de" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45149 } } peers { permanent_uuid: "d295b421f67a4d0e962b6c2365248448" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36957 } } peers { permanent_uuid: "b1109a56871a442584d8a397a4c72542" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38529 } }
I20260926 16:31:31.896159 13709 raft_consensus.cc:374] T 71779fdf45ca4138bb047d75eb576f57 P 93c3a12e5462468b875b5d0fbdd048de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:31:31.896250 13709 raft_consensus.cc:729] T 71779fdf45ca4138bb047d75eb576f57 P 93c3a12e5462468b875b5d0fbdd048de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93c3a12e5462468b875b5d0fbdd048de, State: Initialized, Role: FOLLOWER
I20260926 16:31:31.896955 13709 consensus_queue.cc:260] T 71779fdf45ca4138bb047d75eb576f57 P 93c3a12e5462468b875b5d0fbdd048de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c3a12e5462468b875b5d0fbdd048de" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45149 } } peers { permanent_uuid: "d295b421f67a4d0e962b6c2365248448" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36957 } } peers { permanent_uuid: "b1109a56871a442584d8a397a4c72542" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38529 } }
I20260926 16:31:31.898034 13677 heartbeater.cc:503] Master 127.6.33.62:33939 was elected leader, sending a full tablet report...
I20260926 16:31:31.899430 13710 raft_consensus.cc:348] T 71779fdf45ca4138bb047d75eb576f57 P d295b421f67a4d0e962b6c2365248448 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c3a12e5462468b875b5d0fbdd048de" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45149 } } peers { permanent_uuid: "d295b421f67a4d0e962b6c2365248448" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36957 } } peers { permanent_uuid: "b1109a56871a442584d8a397a4c72542" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38529 } }
I20260926 16:31:31.899837 13710 raft_consensus.cc:374] T 71779fdf45ca4138bb047d75eb576f57 P d295b421f67a4d0e962b6c2365248448 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:31:31.899932 13710 raft_consensus.cc:729] T 71779fdf45ca4138bb047d75eb576f57 P d295b421f67a4d0e962b6c2365248448 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d295b421f67a4d0e962b6c2365248448, State: Initialized, Role: FOLLOWER
I20260926 16:31:31.900112 13709 ts_tablet_manager.cc:1428] T 71779fdf45ca4138bb047d75eb576f57 P 93c3a12e5462468b875b5d0fbdd048de: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260926 16:31:31.900492 13710 consensus_queue.cc:260] T 71779fdf45ca4138bb047d75eb576f57 P d295b421f67a4d0e962b6c2365248448 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c3a12e5462468b875b5d0fbdd048de" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45149 } } peers { permanent_uuid: "d295b421f67a4d0e962b6c2365248448" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36957 } } peers { permanent_uuid: "b1109a56871a442584d8a397a4c72542" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38529 } }
I20260926 16:31:31.902870 13712 raft_consensus.cc:348] T 71779fdf45ca4138bb047d75eb576f57 P b1109a56871a442584d8a397a4c72542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c3a12e5462468b875b5d0fbdd048de" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45149 } } peers { permanent_uuid: "d295b421f67a4d0e962b6c2365248448" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36957 } } peers { permanent_uuid: "b1109a56871a442584d8a397a4c72542" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38529 } }
I20260926 16:31:31.903376 13712 raft_consensus.cc:374] T 71779fdf45ca4138bb047d75eb576f57 P b1109a56871a442584d8a397a4c72542 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:31:31.903484 13712 raft_consensus.cc:729] T 71779fdf45ca4138bb047d75eb576f57 P b1109a56871a442584d8a397a4c72542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1109a56871a442584d8a397a4c72542, State: Initialized, Role: FOLLOWER
I20260926 16:31:31.904855 13710 ts_tablet_manager.cc:1428] T 71779fdf45ca4138bb047d75eb576f57 P d295b421f67a4d0e962b6c2365248448: Time spent starting tablet: real 0.016s user 0.011s sys 0.004s
I20260926 16:31:31.904875 13712 consensus_queue.cc:260] T 71779fdf45ca4138bb047d75eb576f57 P b1109a56871a442584d8a397a4c72542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c3a12e5462468b875b5d0fbdd048de" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45149 } } peers { permanent_uuid: "d295b421f67a4d0e962b6c2365248448" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36957 } } peers { permanent_uuid: "b1109a56871a442584d8a397a4c72542" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38529 } }
I20260926 16:31:31.905860 13536 heartbeater.cc:503] Master 127.6.33.62:33939 was elected leader, sending a full tablet report...
I20260926 16:31:31.909545 13712 ts_tablet_manager.cc:1428] T 71779fdf45ca4138bb047d75eb576f57 P b1109a56871a442584d8a397a4c72542: Time spent starting tablet: real 0.021s user 0.015s sys 0.002s
I20260926 16:31:31.912925 13716 raft_consensus.cc:482] T 71779fdf45ca4138bb047d75eb576f57 P d295b421f67a4d0e962b6c2365248448 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:31:31.913162 13716 raft_consensus.cc:504] T 71779fdf45ca4138bb047d75eb576f57 P d295b421f67a4d0e962b6c2365248448 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c3a12e5462468b875b5d0fbdd048de" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45149 } } peers { permanent_uuid: "d295b421f67a4d0e962b6c2365248448" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36957 } } peers { permanent_uuid: "b1109a56871a442584d8a397a4c72542" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38529 } }
I20260926 16:31:31.914985 13716 leader_election.cc:288] T 71779fdf45ca4138bb047d75eb576f57 P d295b421f67a4d0e962b6c2365248448 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93c3a12e5462468b875b5d0fbdd048de (127.6.33.3:45149), b1109a56871a442584d8a397a4c72542 (127.6.33.1:38529)
I20260926 16:31:31.923002 13704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:31.915413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:60463 (local address 127.6.33.3:45149)
0926 16:31:31.915610 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:31.915618 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:31.916169 (+ 551us) server_negotiation.cc:407] Connection header received
0926 16:31:31.916353 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:31.916361 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:31.916468 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:31.916571 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:31:31.917223 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:31.918213 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:31.920908 (+ 2695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:31.922154 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:31.922297 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:31.922409 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:31.922600 (+ 191us) server_negotiation.cc:299] Negotiation successful
0926 16:31:31.922734 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 16:31:31.923441 13718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:31.915414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.3:45149 (local address 127.6.33.2:60463)
0926 16:31:31.916038 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:31.916062 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:31.916199 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:31.916625 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:31.916635 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:31.916662 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:31:31.917097 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:31.917109 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:31.918451 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:31.918454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:31.920712 (+ 2258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:31.920727 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:31.922334 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:31.922338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:31.922447 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:31.922471 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 16:31:31.922548 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 16:31:31.922635 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"mutex_wait_us":38,"thread_start_us":294,"threads_started":1}
I20260926 16:31:31.923655 13707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:31.916138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:52839 (local address 127.6.33.1:38529)
0926 16:31:31.916318 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:31.916324 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:31.916882 (+ 558us) server_negotiation.cc:407] Connection header received
0926 16:31:31.917086 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:31.917091 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:31.917164 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:31.917225 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:31:31.917768 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:31.918843 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:31.921289 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:31.922679 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:31.922819 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:31.922920 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:31.923225 (+ 305us) server_negotiation.cc:299] Negotiation successful
0926 16:31:31.923357 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 16:31:31.924275 13719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:31.916049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:38529 (local address 127.6.33.2:52839)
0926 16:31:31.916783 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:31.916801 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:31.916920 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:31.917366 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:31.917368 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:31.917394 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:31:31.917626 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:31.917632 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:31.918972 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:31.918977 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:31.921141 (+ 2164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:31.921151 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:31.922897 (+ 1746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:31.922904 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:31.923043 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:31.923072 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 16:31:31.923804 (+ 732us) client_negotiation.cc:240] Negotiation successful
0926 16:31:31.923937 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"mutex_wait_us":24,"thread_start_us":442,"threads_started":1}
I20260926 16:31:31.924551 13631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71779fdf45ca4138bb047d75eb576f57" candidate_uuid: "d295b421f67a4d0e962b6c2365248448" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93c3a12e5462468b875b5d0fbdd048de" is_pre_election: true
I20260926 16:31:31.925072 13631 raft_consensus.cc:2454] T 71779fdf45ca4138bb047d75eb576f57 P 93c3a12e5462468b875b5d0fbdd048de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d295b421f67a4d0e962b6c2365248448 in term 0.
I20260926 16:31:31.925125 13344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71779fdf45ca4138bb047d75eb576f57" candidate_uuid: "d295b421f67a4d0e962b6c2365248448" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1109a56871a442584d8a397a4c72542" is_pre_election: true
I20260926 16:31:31.925578 13344 raft_consensus.cc:2454] T 71779fdf45ca4138bb047d75eb576f57 P b1109a56871a442584d8a397a4c72542 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d295b421f67a4d0e962b6c2365248448 in term 0.
I20260926 16:31:31.925791 13426 leader_election.cc:302] T 71779fdf45ca4138bb047d75eb576f57 P d295b421f67a4d0e962b6c2365248448 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93c3a12e5462468b875b5d0fbdd048de, d295b421f67a4d0e962b6c2365248448; no voters:
I20260926 16:31:31.926187 13716 raft_consensus.cc:2781] T 71779fdf45ca4138bb047d75eb576f57 P d295b421f67a4d0e962b6c2365248448 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:31:31.926327 13716 raft_consensus.cc:482] T 71779fdf45ca4138bb047d75eb576f57 P d295b421f67a4d0e962b6c2365248448 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:31:31.926455 13716 raft_consensus.cc:3037] T 71779fdf45ca4138bb047d75eb576f57 P d295b421f67a4d0e962b6c2365248448 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:31:31.928610 13716 raft_consensus.cc:504] T 71779fdf45ca4138bb047d75eb576f57 P d295b421f67a4d0e962b6c2365248448 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c3a12e5462468b875b5d0fbdd048de" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45149 } } peers { permanent_uuid: "d295b421f67a4d0e962b6c2365248448" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36957 } } peers { permanent_uuid: "b1109a56871a442584d8a397a4c72542" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38529 } }
I20260926 16:31:31.929227 13716 leader_election.cc:288] T 71779fdf45ca4138bb047d75eb576f57 P d295b421f67a4d0e962b6c2365248448 [CANDIDATE]: Term 1 election: Requested vote from peers 93c3a12e5462468b875b5d0fbdd048de (127.6.33.3:45149), b1109a56871a442584d8a397a4c72542 (127.6.33.1:38529)
I20260926 16:31:31.929668 13631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71779fdf45ca4138bb047d75eb576f57" candidate_uuid: "d295b421f67a4d0e962b6c2365248448" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93c3a12e5462468b875b5d0fbdd048de"
I20260926 16:31:31.929814 13344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71779fdf45ca4138bb047d75eb576f57" candidate_uuid: "d295b421f67a4d0e962b6c2365248448" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1109a56871a442584d8a397a4c72542"
I20260926 16:31:31.929894 13631 raft_consensus.cc:3037] T 71779fdf45ca4138bb047d75eb576f57 P 93c3a12e5462468b875b5d0fbdd048de [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:31:31.930034 13344 raft_consensus.cc:3037] T 71779fdf45ca4138bb047d75eb576f57 P b1109a56871a442584d8a397a4c72542 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:31:31.932186 13344 raft_consensus.cc:2454] T 71779fdf45ca4138bb047d75eb576f57 P b1109a56871a442584d8a397a4c72542 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d295b421f67a4d0e962b6c2365248448 in term 1.
I20260926 16:31:31.932459 13631 raft_consensus.cc:2454] T 71779fdf45ca4138bb047d75eb576f57 P 93c3a12e5462468b875b5d0fbdd048de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d295b421f67a4d0e962b6c2365248448 in term 1.
I20260926 16:31:31.932577 13426 leader_election.cc:302] T 71779fdf45ca4138bb047d75eb576f57 P d295b421f67a4d0e962b6c2365248448 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b1109a56871a442584d8a397a4c72542, d295b421f67a4d0e962b6c2365248448; no voters:
I20260926 16:31:31.932916 13716 raft_consensus.cc:2781] T 71779fdf45ca4138bb047d75eb576f57 P d295b421f67a4d0e962b6c2365248448 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:31:31.933336 13716 raft_consensus.cc:686] T 71779fdf45ca4138bb047d75eb576f57 P d295b421f67a4d0e962b6c2365248448 [term 1 LEADER]: Becoming Leader. State: Replica: d295b421f67a4d0e962b6c2365248448, State: Running, Role: LEADER
I20260926 16:31:31.933920 13716 consensus_queue.cc:237] T 71779fdf45ca4138bb047d75eb576f57 P d295b421f67a4d0e962b6c2365248448 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c3a12e5462468b875b5d0fbdd048de" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45149 } } peers { permanent_uuid: "d295b421f67a4d0e962b6c2365248448" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36957 } } peers { permanent_uuid: "b1109a56871a442584d8a397a4c72542" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38529 } }
I20260926 16:31:31.939262 12422 catalog_manager.cc:5495] T 71779fdf45ca4138bb047d75eb576f57 P d295b421f67a4d0e962b6c2365248448 reported cstate change: term changed from 0 to 1, leader changed from <none> to d295b421f67a4d0e962b6c2365248448 (127.6.33.2). New cstate: current_term: 1 leader_uuid: "d295b421f67a4d0e962b6c2365248448" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c3a12e5462468b875b5d0fbdd048de" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d295b421f67a4d0e962b6c2365248448" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36957 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b1109a56871a442584d8a397a4c72542" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38529 } health_report { overall_health: UNKNOWN } } }
I20260926 16:31:31.980939 13706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:31.975559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40194 (local address 127.6.33.2:36957)
0926 16:31:31.975759 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:31.975766 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:31.975785 (+ 19us) server_negotiation.cc:407] Connection header received
0926 16:31:31.976039 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:31.976042 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:31.976124 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:31.976275 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:31:31.976614 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:31.977342 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:31.978706 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:31.979099 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:31.979225 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:31.979703 (+ 478us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:31:31.980285 (+ 582us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:31:31.980508 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:31.980566 (+ 58us) server_negotiation.cc:299] Negotiation successful
0926 16:31:31.980641 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260926 16:31:31.988957 13392 tablet.cc:2369] T 71779fdf45ca4138bb047d75eb576f57 P b1109a56871a442584d8a397a4c72542: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:31:31.989470 13631 raft_consensus.cc:1261] T 71779fdf45ca4138bb047d75eb576f57 P 93c3a12e5462468b875b5d0fbdd048de [term 1 FOLLOWER]: Refusing update from remote peer d295b421f67a4d0e962b6c2365248448: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:31:31.989588 13344 raft_consensus.cc:1261] T 71779fdf45ca4138bb047d75eb576f57 P b1109a56871a442584d8a397a4c72542 [term 1 FOLLOWER]: Refusing update from remote peer d295b421f67a4d0e962b6c2365248448: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:31:31.990427 13716 consensus_queue.cc:1035] T 71779fdf45ca4138bb047d75eb576f57 P d295b421f67a4d0e962b6c2365248448 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93c3a12e5462468b875b5d0fbdd048de" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:31:31.990770 13720 consensus_queue.cc:1035] T 71779fdf45ca4138bb047d75eb576f57 P d295b421f67a4d0e962b6c2365248448 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1109a56871a442584d8a397a4c72542" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 16:31:32.000594 13686 tablet.cc:2369] T 71779fdf45ca4138bb047d75eb576f57 P 93c3a12e5462468b875b5d0fbdd048de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:31:32.004961 13723 mvcc.cc:205] Tried to move back new op lower bound from 7333643435978137600 to 7333643435764928512. Current Snapshot: MvccSnapshot[applied={T|T < 7333643435978137600}]
I20260926 16:31:32.012416 13725 mvcc.cc:205] Tried to move back new op lower bound from 7333643435978137600 to 7333643435764928512. Current Snapshot: MvccSnapshot[applied={T|T < 7333643435978137600}]
I20260926 16:31:32.029737 12422 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:31:32.030053 12422 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:31:32.037638 12422 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7a7973add6e4409dabfab1a4d9e6b175: Sending DeleteTablet for 3 replicas of tablet 71779fdf45ca4138bb047d75eb576f57
I20260926 16:31:32.038822 13611 tablet_service.cc:1514] Processing DeleteTablet for tablet 71779fdf45ca4138bb047d75eb576f57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:31:32 UTC) from {username='oryx', principal='oryx/127.6.33.62@KRBTEST.COM'} at 127.0.0.1:58622
I20260926 16:31:32.038889 13470 tablet_service.cc:1514] Processing DeleteTablet for tablet 71779fdf45ca4138bb047d75eb576f57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:31:32 UTC) from {username='oryx', principal='oryx/127.6.33.62@KRBTEST.COM'} at 127.0.0.1:40180
I20260926 16:31:32.041711 13324 tablet_service.cc:1514] Processing DeleteTablet for tablet 71779fdf45ca4138bb047d75eb576f57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:31:32 UTC) from {username='oryx', principal='oryx/127.6.33.62@KRBTEST.COM'} at 127.0.0.1:39916
I20260926 16:31:32.044135 13734 tablet_replica.cc:330] T 71779fdf45ca4138bb047d75eb576f57 P b1109a56871a442584d8a397a4c72542: stopping tablet replica
I20260926 16:31:32.045065 13734 raft_consensus.cc:2229] T 71779fdf45ca4138bb047d75eb576f57 P b1109a56871a442584d8a397a4c72542 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:31:32.045773 13734 raft_consensus.cc:2258] T 71779fdf45ca4138bb047d75eb576f57 P b1109a56871a442584d8a397a4c72542 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:31:32.046645 13732 tablet_replica.cc:330] T 71779fdf45ca4138bb047d75eb576f57 P 93c3a12e5462468b875b5d0fbdd048de: stopping tablet replica
I20260926 16:31:32.047057 13732 raft_consensus.cc:2229] T 71779fdf45ca4138bb047d75eb576f57 P 93c3a12e5462468b875b5d0fbdd048de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:31:32.047482 13732 raft_consensus.cc:2258] T 71779fdf45ca4138bb047d75eb576f57 P 93c3a12e5462468b875b5d0fbdd048de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:31:32.049891 13734 ts_tablet_manager.cc:1905] T 71779fdf45ca4138bb047d75eb576f57 P b1109a56871a442584d8a397a4c72542: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:31:32.049891 13732 ts_tablet_manager.cc:1905] T 71779fdf45ca4138bb047d75eb576f57 P 93c3a12e5462468b875b5d0fbdd048de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:31:32.057170 13732 ts_tablet_manager.cc:1918] T 71779fdf45ca4138bb047d75eb576f57 P 93c3a12e5462468b875b5d0fbdd048de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:31:32.057556 13732 log.cc:1196] T 71779fdf45ca4138bb047d75eb576f57 P 93c3a12e5462468b875b5d0fbdd048de: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-2/wal/wals/71779fdf45ca4138bb047d75eb576f57
I20260926 16:31:32.058295 13732 ts_tablet_manager.cc:1939] T 71779fdf45ca4138bb047d75eb576f57 P 93c3a12e5462468b875b5d0fbdd048de: Deleting consensus metadata
I20260926 16:31:32.058445 13733 tablet_replica.cc:330] T 71779fdf45ca4138bb047d75eb576f57 P d295b421f67a4d0e962b6c2365248448: stopping tablet replica
I20260926 16:31:32.059235 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 12390
I20260926 16:31:32.059347 13734 ts_tablet_manager.cc:1918] T 71779fdf45ca4138bb047d75eb576f57 P b1109a56871a442584d8a397a4c72542: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:31:32.060348 13733 raft_consensus.cc:2229] T 71779fdf45ca4138bb047d75eb576f57 P d295b421f67a4d0e962b6c2365248448 [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:31:32.060484 12452 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:31:32.061221 13733 raft_consensus.cc:2258] T 71779fdf45ca4138bb047d75eb576f57 P d295b421f67a4d0e962b6c2365248448 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:31:32.059660 13734 log.cc:1196] T 71779fdf45ca4138bb047d75eb576f57 P b1109a56871a442584d8a397a4c72542: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-0/wal/wals/71779fdf45ca4138bb047d75eb576f57
I20260926 16:31:32.063299 13734 ts_tablet_manager.cc:1939] T 71779fdf45ca4138bb047d75eb576f57 P b1109a56871a442584d8a397a4c72542: Deleting consensus metadata
I20260926 16:31:32.067551 13733 ts_tablet_manager.cc:1905] T 71779fdf45ca4138bb047d75eb576f57 P d295b421f67a4d0e962b6c2365248448: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:31:32.077556 13733 ts_tablet_manager.cc:1918] T 71779fdf45ca4138bb047d75eb576f57 P d295b421f67a4d0e962b6c2365248448: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:31:32.077751 13733 log.cc:1196] T 71779fdf45ca4138bb047d75eb576f57 P d295b421f67a4d0e962b6c2365248448: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-1/wal/wals/71779fdf45ca4138bb047d75eb576f57
I20260926 16:31:32.078447 13733 ts_tablet_manager.cc:1939] T 71779fdf45ca4138bb047d75eb576f57 P d295b421f67a4d0e962b6c2365248448: Deleting consensus metadata
I20260926 16:31:32.083120 13684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:32.044100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57158 (local address 127.6.33.61:37487)
0926 16:31:32.044321 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:32.044331 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:32.057547 (+ 13216us) server_negotiation.cc:407] Connection header received
0926 16:31:32.059776 (+ 2229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:32.059783 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:32.059876 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:32.059985 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:32.062632 (+ 2647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:32.064114 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:32.066264 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:32.066991 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:32.067155 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:32.071535 (+ 4380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:32.071543 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:32.071550 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:32.071603 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:32.075871 (+ 4268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:32.076884 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:32.076892 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:32.076896 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:32.077054 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:32.081270 (+ 4216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:32.081276 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:32.081280 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:32.081701 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:32.081998 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:32.082486 (+ 488us) server_negotiation.cc:299] Negotiation successful
0926 16:31:32.082739 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 16:31:32.102645 13683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:32.057835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46230 (local address 127.6.33.60:42953)
0926 16:31:32.058050 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:32.058056 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:32.083153 (+ 25097us) server_negotiation.cc:407] Connection header received
0926 16:31:32.083450 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:32.083456 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:32.083564 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:32.083678 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:32.086016 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:32.087143 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:32.089429 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:32.090116 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:32.090218 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:32.094615 (+ 4397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:32.094621 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:32.094626 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:32.094669 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:32.099004 (+ 4335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:32.099911 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:32.099917 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:32.099922 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:32.100099 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:32.100531 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:32.100537 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:32.100541 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:32.100978 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:32.101234 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:32.101926 (+ 692us) server_negotiation.cc:299] Negotiation successful
0926 16:31:32.102226 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 16:31:32.256963 12409 catalog_manager.cc:4841] TS 93c3a12e5462468b875b5d0fbdd048de (127.6.33.3:45149): tablet 71779fdf45ca4138bb047d75eb576f57 (table test-table [id=ffe04a9bfaa54487a41d8374855aea26]) successfully deleted
W20260926 16:31:32.257246 12409 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.6.33.3:45149, user_credentials={real_user=slave}} blocked reactor thread for 189700us
I20260926 16:31:32.257426 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 12390
W20260926 16:31:32.279593 13277 connection.cc:535] server connection from 127.0.0.1:39916 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 16:31:32.310499 13683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:32.284934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46244 (local address 127.6.33.60:42953)
0926 16:31:32.285188 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:32.285194 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:32.291899 (+ 6705us) server_negotiation.cc:407] Connection header received
0926 16:31:32.292956 (+ 1057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:32.292966 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:32.293084 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:32.293182 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:32.296144 (+ 2962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:32.297377 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:32.299476 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:32.300141 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:32.300269 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:32.303983 (+ 3714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:32.303991 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:32.303995 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:32.304042 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:32.307123 (+ 3081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:32.308049 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:32.308055 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:32.308059 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:32.308181 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:32.308606 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:32.308611 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:32.308615 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:32.309044 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:32.309283 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:32.310073 (+ 790us) server_negotiation.cc:299] Negotiation successful
0926 16:31:32.310267 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260926 16:31:32.311581 13684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:32.284360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57168 (local address 127.6.33.61:37487)
0926 16:31:32.284588 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:32.284595 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:32.295515 (+ 10920us) server_negotiation.cc:407] Connection header received
0926 16:31:32.295685 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:32.295691 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:32.295770 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:32.295879 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:32.298345 (+ 2466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:32.299486 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:32.301314 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:32.301734 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:32.301876 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:32.304826 (+ 2950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:32.304834 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:32.304839 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:32.304881 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:32.308263 (+ 3382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:32.309057 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:32.309061 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:32.309064 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:32.309224 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:32.309620 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:32.309625 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:32.309629 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:32.310027 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:32.310266 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:32.311013 (+ 747us) server_negotiation.cc:299] Negotiation successful
0926 16:31:32.311234 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 16:31:33.012796 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:33.011925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:33939 (local address 127.6.33.2:56319)
0926 16:31:33.012433 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:33.012582 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:33939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":223,"threads_started":1}
W20260926 16:31:33.013211 13536 heartbeater.cc:644] Failed to heartbeat to 127.6.33.62:33939 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.33.62:33939: connect: Connection refused (error 111)
I20260926 16:31:33.021754 13745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:33.020940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:33939 (local address 127.6.33.1:58169)
0926 16:31:33.021415 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:33.021554 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:33939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":240,"threads_started":1}
W20260926 16:31:33.022372 13390 heartbeater.cc:644] Failed to heartbeat to 127.6.33.62:33939 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.33.62:33939: connect: Connection refused (error 111)
I20260926 16:31:33.031056 13746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:33.030173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:33939 (local address 127.6.33.3:50501)
0926 16:31:33.030638 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:33.030796 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:33939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":258,"threads_started":1}
W20260926 16:31:33.031432 13677 heartbeater.cc:644] Failed to heartbeat to 127.6.33.62:33939 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.33.62:33939: connect: Connection refused (error 111)
I20260926 16:31:33.537945 13747 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a7973add6e4409dabfab1a4d9e6b175)
I20260926 16:31:33.538313 13747 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7a7973add6e4409dabfab1a4d9e6b175" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 33939 } } peers { permanent_uuid: "9d1dfe3b814a4549aebd349a75850ef4" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 37487 } attrs { promote: false } } peers { permanent_uuid: "8b9d3065bcd44983a97ac74155813cdc" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 42953 } attrs { promote: false } }
I20260926 16:31:33.540091 13747 leader_election.cc:288] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a7973add6e4409dabfab1a4d9e6b175 (127.6.33.62:33939), 8b9d3065bcd44983a97ac74155813cdc (127.6.33.60:42953)
I20260926 16:31:33.541275 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:33.540219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:33939 (local address 127.0.0.1:50560)
0926 16:31:33.540809 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:33.540981 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:33939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"threads_started":1}
W20260926 16:31:33.542595 13007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.33.62:33939: connect: Connection refused (error 111)
I20260926 16:31:33.542824 13751 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b9d3065bcd44983a97ac74155813cdc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a7973add6e4409dabfab1a4d9e6b175)
I20260926 16:31:33.543123 13751 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b9d3065bcd44983a97ac74155813cdc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7a7973add6e4409dabfab1a4d9e6b175" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 33939 } } peers { permanent_uuid: "9d1dfe3b814a4549aebd349a75850ef4" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 37487 } attrs { promote: false } } peers { permanent_uuid: "8b9d3065bcd44983a97ac74155813cdc" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 42953 } attrs { promote: false } }
I20260926 16:31:33.544363 13748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:33.543923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:33939 (local address 127.0.0.1:50568)
0926 16:31:33.544083 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:33.544207 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:33939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 16:31:33.544596 13007 leader_election.cc:334] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7a7973add6e4409dabfab1a4d9e6b175 (127.6.33.62:33939): Network error: Client connection negotiation failed: client connection to 127.6.33.62:33939: connect: Connection refused (error 111)
I20260926 16:31:33.544924 13751 leader_election.cc:288] T 00000000000000000000000000000000 P 8b9d3065bcd44983a97ac74155813cdc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a7973add6e4409dabfab1a4d9e6b175 (127.6.33.62:33939), 9d1dfe3b814a4549aebd349a75850ef4 (127.6.33.61:37487)
I20260926 16:31:33.546663 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:33.544979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:33939 (local address 127.0.0.1:50580)
0926 16:31:33.546173 (+ 1194us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:33.546363 (+ 190us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:33939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1072,"thread_start_us":942,"threads_started":1}
W20260926 16:31:33.546939 13196 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.33.62:33939: connect: Connection refused (error 111)
I20260926 16:31:33.549232 13754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:33.548628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:33939 (local address 127.0.0.1:50588)
0926 16:31:33.548847 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:33.548982 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:33939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260926 16:31:33.549408 13196 leader_election.cc:334] T 00000000000000000000000000000000 P 8b9d3065bcd44983a97ac74155813cdc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7a7973add6e4409dabfab1a4d9e6b175 (127.6.33.62:33939): Network error: Client connection negotiation failed: client connection to 127.6.33.62:33939: connect: Connection refused (error 111)
I20260926 16:31:33.550200 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:33.540272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:42953 (local address 127.0.0.1:46254)
0926 16:31:33.541667 (+ 1395us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:33.541691 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:33.541858 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:33.543306 (+ 1448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:33.543312 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:33.543345 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:31:33.543613 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:33.543625 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:33.545246 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:33.545252 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:33.547387 (+ 2135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:33.547407 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:33.549404 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:33.549410 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:33.549601 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:33.549645 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 16:31:33.549730 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 16:31:33.549872 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1272,"thread_start_us":426,"threads_started":1}
I20260926 16:31:33.550524 13750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:33.540532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46254 (local address 127.6.33.60:42953)
0926 16:31:33.542930 (+ 2398us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:33.542937 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:33.542959 (+ 22us) server_negotiation.cc:407] Connection header received
0926 16:31:33.543026 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:33.543032 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:33.543138 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:33.543755 (+ 617us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:31:33.543986 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:33.545121 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:33.547565 (+ 2444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:33.549223 (+ 1658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:33.549355 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:33.549467 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:33.549825 (+ 358us) server_negotiation.cc:299] Negotiation successful
0926 16:31:33.550039 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":2267,"thread_start_us":2140,"threads_started":1}
I20260926 16:31:33.551100 13219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9d1dfe3b814a4549aebd349a75850ef4" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8b9d3065bcd44983a97ac74155813cdc" is_pre_election: true
I20260926 16:31:33.551461 13219 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8b9d3065bcd44983a97ac74155813cdc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d1dfe3b814a4549aebd349a75850ef4 in term 1.
I20260926 16:31:33.551986 13004 leader_election.cc:302] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8b9d3065bcd44983a97ac74155813cdc, 9d1dfe3b814a4549aebd349a75850ef4; no voters: 7a7973add6e4409dabfab1a4d9e6b175
I20260926 16:31:33.552479 13747 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 16:31:33.552608 13747 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7a7973add6e4409dabfab1a4d9e6b175)
I20260926 16:31:33.552711 13747 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4 [term 1 FOLLOWER]: Advancing to term 2
I20260926 16:31:33.553730 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:33.545099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.61:37487 (local address 127.0.0.1:57178)
0926 16:31:33.546872 (+ 1773us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:33.546889 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:33.547007 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:33.548006 (+ 999us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:33.548011 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:33.548043 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:31:33.548415 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:33.548429 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:33.549675 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:33.549680 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:33.551438 (+ 1758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:33.551458 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:33.553161 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:33.553166 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:33.553291 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:33.553316 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 16:31:33.553398 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 16:31:33.553502 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1669,"mutex_wait_us":1,"thread_start_us":2111,"threads_started":1}
I20260926 16:31:33.553982 13755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:33.545277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57178 (local address 127.6.33.61:37487)
0926 16:31:33.547692 (+ 2415us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:33.547699 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:33.547721 (+ 22us) server_negotiation.cc:407] Connection header received
0926 16:31:33.547789 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:33.547796 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:33.547871 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:33.547971 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:31:33.548595 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:33.549549 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:33.551623 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:33.552988 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:33.553146 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:33.553263 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:33.553446 (+ 183us) server_negotiation.cc:299] Negotiation successful
0926 16:31:33.553589 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2255,"thread_start_us":2051,"threads_started":1}
I20260926 16:31:33.554682 13029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8b9d3065bcd44983a97ac74155813cdc" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9d1dfe3b814a4549aebd349a75850ef4" is_pre_election: true
I20260926 16:31:33.555631 13747 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7a7973add6e4409dabfab1a4d9e6b175" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 33939 } } peers { permanent_uuid: "9d1dfe3b814a4549aebd349a75850ef4" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 37487 } attrs { promote: false } } peers { permanent_uuid: "8b9d3065bcd44983a97ac74155813cdc" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 42953 } attrs { promote: false } }
I20260926 16:31:33.556089 13029 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8b9d3065bcd44983a97ac74155813cdc in current term 2: Already voted for candidate 9d1dfe3b814a4549aebd349a75850ef4 in this term.
I20260926 16:31:33.556741 13747 leader_election.cc:288] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4 [CANDIDATE]: Term 2 election: Requested vote from peers 7a7973add6e4409dabfab1a4d9e6b175 (127.6.33.62:33939), 8b9d3065bcd44983a97ac74155813cdc (127.6.33.60:42953)
I20260926 16:31:33.556947 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:33.556407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:33939 (local address 127.0.0.1:50590)
0926 16:31:33.556596 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:33.556687 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:33939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 16:31:33.557322 13219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9d1dfe3b814a4549aebd349a75850ef4" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8b9d3065bcd44983a97ac74155813cdc"
I20260926 16:31:33.557610 13219 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b9d3065bcd44983a97ac74155813cdc [term 1 FOLLOWER]: Advancing to term 2
I20260926 16:31:33.557806 13195 leader_election.cc:302] T 00000000000000000000000000000000 P 8b9d3065bcd44983a97ac74155813cdc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b9d3065bcd44983a97ac74155813cdc; no voters: 7a7973add6e4409dabfab1a4d9e6b175, 9d1dfe3b814a4549aebd349a75850ef4
I20260926 16:31:33.557942 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:33.557549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:33939 (local address 127.0.0.1:50598)
0926 16:31:33.557708 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:33.557809 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:33939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 16:31:33.558132 13007 leader_election.cc:334] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 7a7973add6e4409dabfab1a4d9e6b175 (127.6.33.62:33939): Network error: Client connection negotiation failed: client connection to 127.6.33.62:33939: connect: Connection refused (error 111)
I20260926 16:31:33.560259 13219 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8b9d3065bcd44983a97ac74155813cdc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d1dfe3b814a4549aebd349a75850ef4 in term 2.
I20260926 16:31:33.560513 13751 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 8b9d3065bcd44983a97ac74155813cdc [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 16:31:33.560842 13004 leader_election.cc:302] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8b9d3065bcd44983a97ac74155813cdc, 9d1dfe3b814a4549aebd349a75850ef4; no voters: 7a7973add6e4409dabfab1a4d9e6b175
I20260926 16:31:33.561189 13747 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 16:31:33.561648 13747 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4 [term 2 LEADER]: Becoming Leader. State: Replica: 9d1dfe3b814a4549aebd349a75850ef4, State: Running, Role: LEADER
I20260926 16:31:33.562048 13747 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7a7973add6e4409dabfab1a4d9e6b175" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 33939 } } peers { permanent_uuid: "9d1dfe3b814a4549aebd349a75850ef4" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 37487 } attrs { promote: false } } peers { permanent_uuid: "8b9d3065bcd44983a97ac74155813cdc" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 42953 } attrs { promote: false } }
I20260926 16:31:33.563913 13757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d1dfe3b814a4549aebd349a75850ef4. Latest consensus state: current_term: 2 leader_uuid: "9d1dfe3b814a4549aebd349a75850ef4" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7a7973add6e4409dabfab1a4d9e6b175" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 33939 } } peers { permanent_uuid: "9d1dfe3b814a4549aebd349a75850ef4" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 37487 } attrs { promote: false } } peers { permanent_uuid: "8b9d3065bcd44983a97ac74155813cdc" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 42953 } attrs { promote: false } } }
I20260926 16:31:33.564334 13757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4 [sys.catalog]: This master's current role is: LEADER
I20260926 16:31:33.565089 13759 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:31:33.568673 13759 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:31:33.569635 13759 catalog_manager.cc:1259] Loaded cluster ID: cb13d77c4ee3430abf37211ff83e1ee5
I20260926 16:31:33.569707 13759 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:31:33.570955 13759 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:31:33.571779 13759 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4: Loaded TSK: 0
I20260926 16:31:33.572666 13759 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:31:33.600567 13020 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57168:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:31:33.607573 13020 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:31:33.611847 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:33.611439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:33939 (local address 127.0.0.1:50628)
0926 16:31:33.611630 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:33.611704 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:33939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 16:31:33.611920 13219 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8b9d3065bcd44983a97ac74155813cdc [term 2 FOLLOWER]: Refusing update from remote peer 9d1dfe3b814a4549aebd349a75850ef4: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260926 16:31:33.612694 13757 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b9d3065bcd44983a97ac74155813cdc" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 42953 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260926 16:31:33.612895 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:33.612502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:33939 (local address 127.0.0.1:50644)
0926 16:31:33.612651 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:33.612727 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:33939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 16:31:33.613089 13007 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4 -> Peer 7a7973add6e4409dabfab1a4d9e6b175 (127.6.33.62:33939): Couldn't send request to peer 7a7973add6e4409dabfab1a4d9e6b175. Status: Network error: Client connection negotiation failed: client connection to 127.6.33.62:33939: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 16:31:33.615535 13751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b9d3065bcd44983a97ac74155813cdc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d1dfe3b814a4549aebd349a75850ef4. Latest consensus state: current_term: 2 leader_uuid: "9d1dfe3b814a4549aebd349a75850ef4" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7a7973add6e4409dabfab1a4d9e6b175" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 33939 } } peers { permanent_uuid: "9d1dfe3b814a4549aebd349a75850ef4" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 37487 } attrs { promote: false } } peers { permanent_uuid: "8b9d3065bcd44983a97ac74155813cdc" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 42953 } attrs { promote: false } } }
I20260926 16:31:33.615828 13751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b9d3065bcd44983a97ac74155813cdc [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:31:33.616844 13760 mvcc.cc:205] Tried to move back new op lower bound from 7333643442627792896 to 7333643442433490944. Current Snapshot: MvccSnapshot[applied={T|T < 7333643442627792896}]
I20260926 16:31:33.617369 13747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "9d1dfe3b814a4549aebd349a75850ef4" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7a7973add6e4409dabfab1a4d9e6b175" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 33939 } } peers { permanent_uuid: "9d1dfe3b814a4549aebd349a75850ef4" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 37487 } attrs { promote: false } } peers { permanent_uuid: "8b9d3065bcd44983a97ac74155813cdc" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 42953 } attrs { promote: false } } }
I20260926 16:31:33.617699 13747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4 [sys.catalog]: This master's current role is: LEADER
I20260926 16:31:33.617751 13751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b9d3065bcd44983a97ac74155813cdc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "9d1dfe3b814a4549aebd349a75850ef4" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7a7973add6e4409dabfab1a4d9e6b175" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 33939 } } peers { permanent_uuid: "9d1dfe3b814a4549aebd349a75850ef4" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 37487 } attrs { promote: false } } peers { permanent_uuid: "8b9d3065bcd44983a97ac74155813cdc" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 42953 } attrs { promote: false } } }
I20260926 16:31:33.618027 13751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b9d3065bcd44983a97ac74155813cdc [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:31:33.638616 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:33.630759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.2:36957 (local address 127.0.0.1:40200)
0926 16:31:33.631339 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:33.631357 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:33.631474 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:33.632735 (+ 1261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:33.632739 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:33.632788 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:31:33.633120 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:33.633131 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:33.634065 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:33.634073 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:33.635869 (+ 1796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:33.635883 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:33.637722 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:33.637728 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:33.637933 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:33.637971 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 16:31:33.638055 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 16:31:33.638184 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"mutex_wait_us":19,"thread_start_us":583,"threads_started":1}
I20260926 16:31:33.639230 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:33.629590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.3:45149 (local address 127.0.0.1:58636)
0926 16:31:33.630128 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:33.630263 (+ 135us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:33.630650 (+ 387us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:33.632304 (+ 1654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:33.632312 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:33.632350 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:31:33.632609 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:33.632623 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:33.634925 (+ 2302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:33.634931 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:33.636860 (+ 1929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:33.636875 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:33.638463 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:33.638468 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:33.638679 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:33.638713 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 16:31:33.638803 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 16:31:33.638954 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":28}
I20260926 16:31:33.639256 13769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:33.631920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40200 (local address 127.6.33.2:36957)
0926 16:31:33.632481 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:33.632485 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:33.632500 (+ 15us) server_negotiation.cc:407] Connection header received
0926 16:31:33.632553 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:33.632556 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:33.632608 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:33.632722 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:31:33.633260 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:33.633942 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:33.636053 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:33.637336 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:33.637462 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:33.637563 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:33.638821 (+ 1258us) server_negotiation.cc:299] Negotiation successful
0926 16:31:33.638959 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":1153,"threads_started":1}
I20260926 16:31:33.639721 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:33.630052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58636 (local address 127.6.33.3:45149)
0926 16:31:33.631176 (+ 1124us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:33.631182 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:33.631202 (+ 20us) server_negotiation.cc:407] Connection header received
0926 16:31:33.631276 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:33.631282 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:33.631369 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:33.631445 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:31:33.633261 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:33.634754 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:33.637027 (+ 2273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:33.638302 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:33.638438 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:33.638545 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:33.639424 (+ 879us) server_negotiation.cc:299] Negotiation successful
0926 16:31:33.639537 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":979,"thread_start_us":1746,"threads_started":1}
I20260926 16:31:33.639992 13470 tablet_service.cc:1467] Processing CreateTablet for tablet b9ba1366189c404080acbd0b54ba17eb (DEFAULT_TABLE table=test-table [id=468031f9ed074bc984efcaac97283326]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:31:33.640420 13470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9ba1366189c404080acbd0b54ba17eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:31:33.641417 13611 tablet_service.cc:1467] Processing CreateTablet for tablet b9ba1366189c404080acbd0b54ba17eb (DEFAULT_TABLE table=test-table [id=468031f9ed074bc984efcaac97283326]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:31:33.641819 13611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9ba1366189c404080acbd0b54ba17eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:31:33.643150 13748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:33.630474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:38529 (local address 127.0.0.1:39920)
0926 16:31:33.631072 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:33.631089 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:33.631192 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:33.635511 (+ 4319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:33.635516 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:33.635543 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:31:33.635863 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:33.635877 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:33.636885 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:33.636891 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:33.639117 (+ 2226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:33.639128 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:33.642448 (+ 3320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:33.642456 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:33.642629 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:33.642663 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 16:31:33.642728 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 16:31:33.642855 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"mutex_wait_us":148}
I20260926 16:31:33.643926 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:33.630186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39920 (local address 127.6.33.1:38529)
0926 16:31:33.635230 (+ 5044us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:33.635235 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:33.635250 (+ 15us) server_negotiation.cc:407] Connection header received
0926 16:31:33.635301 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:33.635305 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:33.635359 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:33.635499 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:31:33.636012 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:33.636689 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:33.639261 (+ 2572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:33.642165 (+ 2904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:33.643357 (+ 1192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:33.643480 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:33.643533 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 16:31:33.643656 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4933,"thread_start_us":4844,"threads_started":1}
I20260926 16:31:33.644809 13324 tablet_service.cc:1467] Processing CreateTablet for tablet b9ba1366189c404080acbd0b54ba17eb (DEFAULT_TABLE table=test-table [id=468031f9ed074bc984efcaac97283326]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:31:33.645227 13324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9ba1366189c404080acbd0b54ba17eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:31:33.648733 13771 tablet_bootstrap.cc:492] T b9ba1366189c404080acbd0b54ba17eb P 93c3a12e5462468b875b5d0fbdd048de: Bootstrap starting.
I20260926 16:31:33.648932 13770 tablet_bootstrap.cc:492] T b9ba1366189c404080acbd0b54ba17eb P d295b421f67a4d0e962b6c2365248448: Bootstrap starting.
I20260926 16:31:33.650115 13771 tablet_bootstrap.cc:654] T b9ba1366189c404080acbd0b54ba17eb P 93c3a12e5462468b875b5d0fbdd048de: Neither blocks nor log segments found. Creating new log.
I20260926 16:31:33.650628 13770 tablet_bootstrap.cc:654] T b9ba1366189c404080acbd0b54ba17eb P d295b421f67a4d0e962b6c2365248448: Neither blocks nor log segments found. Creating new log.
I20260926 16:31:33.652112 13771 tablet_bootstrap.cc:492] T b9ba1366189c404080acbd0b54ba17eb P 93c3a12e5462468b875b5d0fbdd048de: No bootstrap required, opened a new log
I20260926 16:31:33.652151 13774 tablet_bootstrap.cc:492] T b9ba1366189c404080acbd0b54ba17eb P b1109a56871a442584d8a397a4c72542: Bootstrap starting.
I20260926 16:31:33.652257 13771 ts_tablet_manager.cc:1397] T b9ba1366189c404080acbd0b54ba17eb P 93c3a12e5462468b875b5d0fbdd048de: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260926 16:31:33.653086 13771 raft_consensus.cc:348] T b9ba1366189c404080acbd0b54ba17eb P 93c3a12e5462468b875b5d0fbdd048de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c3a12e5462468b875b5d0fbdd048de" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45149 } } peers { permanent_uuid: "b1109a56871a442584d8a397a4c72542" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38529 } } peers { permanent_uuid: "d295b421f67a4d0e962b6c2365248448" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36957 } }
I20260926 16:31:33.653342 13771 raft_consensus.cc:374] T b9ba1366189c404080acbd0b54ba17eb P 93c3a12e5462468b875b5d0fbdd048de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:31:33.653450 13771 raft_consensus.cc:729] T b9ba1366189c404080acbd0b54ba17eb P 93c3a12e5462468b875b5d0fbdd048de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93c3a12e5462468b875b5d0fbdd048de, State: Initialized, Role: FOLLOWER
I20260926 16:31:33.653676 13771 consensus_queue.cc:260] T b9ba1366189c404080acbd0b54ba17eb P 93c3a12e5462468b875b5d0fbdd048de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c3a12e5462468b875b5d0fbdd048de" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45149 } } peers { permanent_uuid: "b1109a56871a442584d8a397a4c72542" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38529 } } peers { permanent_uuid: "d295b421f67a4d0e962b6c2365248448" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36957 } }
I20260926 16:31:33.654798 13774 tablet_bootstrap.cc:654] T b9ba1366189c404080acbd0b54ba17eb P b1109a56871a442584d8a397a4c72542: Neither blocks nor log segments found. Creating new log.
I20260926 16:31:33.656569 13771 ts_tablet_manager.cc:1428] T b9ba1366189c404080acbd0b54ba17eb P 93c3a12e5462468b875b5d0fbdd048de: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
I20260926 16:31:33.660403 13678 heartbeater.cc:503] Master 127.6.33.61:37487 was elected leader, sending a full tablet report...
I20260926 16:31:33.660818 13774 tablet_bootstrap.cc:492] T b9ba1366189c404080acbd0b54ba17eb P b1109a56871a442584d8a397a4c72542: No bootstrap required, opened a new log
I20260926 16:31:33.660993 13774 ts_tablet_manager.cc:1397] T b9ba1366189c404080acbd0b54ba17eb P b1109a56871a442584d8a397a4c72542: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260926 16:31:33.661198 13777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:33.657367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:33939 (local address 127.6.33.3:34525)
0926 16:31:33.660789 (+ 3422us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:33.660932 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:33939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3252,"threads_started":1}
I20260926 16:31:33.661849 13774 raft_consensus.cc:348] T b9ba1366189c404080acbd0b54ba17eb P b1109a56871a442584d8a397a4c72542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c3a12e5462468b875b5d0fbdd048de" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45149 } } peers { permanent_uuid: "b1109a56871a442584d8a397a4c72542" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38529 } } peers { permanent_uuid: "d295b421f67a4d0e962b6c2365248448" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36957 } }
I20260926 16:31:33.662113 13774 raft_consensus.cc:374] T b9ba1366189c404080acbd0b54ba17eb P b1109a56871a442584d8a397a4c72542 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:31:33.662221 13774 raft_consensus.cc:729] T b9ba1366189c404080acbd0b54ba17eb P b1109a56871a442584d8a397a4c72542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1109a56871a442584d8a397a4c72542, State: Initialized, Role: FOLLOWER
I20260926 16:31:33.662513 13774 consensus_queue.cc:260] T b9ba1366189c404080acbd0b54ba17eb P b1109a56871a442584d8a397a4c72542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c3a12e5462468b875b5d0fbdd048de" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45149 } } peers { permanent_uuid: "b1109a56871a442584d8a397a4c72542" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38529 } } peers { permanent_uuid: "d295b421f67a4d0e962b6c2365248448" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36957 } }
I20260926 16:31:33.663087 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:33.661665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:33939 (local address 127.6.33.3:58613)
0926 16:31:33.662278 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:33.662829 (+ 551us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:33939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":273,"threads_started":1}
I20260926 16:31:33.663358 13774 ts_tablet_manager.cc:1428] T b9ba1366189c404080acbd0b54ba17eb P b1109a56871a442584d8a397a4c72542: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 16:31:33.663707 13770 tablet_bootstrap.cc:492] T b9ba1366189c404080acbd0b54ba17eb P d295b421f67a4d0e962b6c2365248448: No bootstrap required, opened a new log
I20260926 16:31:33.663863 13770 ts_tablet_manager.cc:1397] T b9ba1366189c404080acbd0b54ba17eb P d295b421f67a4d0e962b6c2365248448: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.011s
I20260926 16:31:33.664954 13770 raft_consensus.cc:348] T b9ba1366189c404080acbd0b54ba17eb P d295b421f67a4d0e962b6c2365248448 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c3a12e5462468b875b5d0fbdd048de" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45149 } } peers { permanent_uuid: "b1109a56871a442584d8a397a4c72542" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38529 } } peers { permanent_uuid: "d295b421f67a4d0e962b6c2365248448" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36957 } }
I20260926 16:31:33.665249 13770 raft_consensus.cc:374] T b9ba1366189c404080acbd0b54ba17eb P d295b421f67a4d0e962b6c2365248448 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:31:33.665343 13770 raft_consensus.cc:729] T b9ba1366189c404080acbd0b54ba17eb P d295b421f67a4d0e962b6c2365248448 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d295b421f67a4d0e962b6c2365248448, State: Initialized, Role: FOLLOWER
I20260926 16:31:33.665743 13391 heartbeater.cc:503] Master 127.6.33.61:37487 was elected leader, sending a full tablet report...
I20260926 16:31:33.665601 13770 consensus_queue.cc:260] T b9ba1366189c404080acbd0b54ba17eb P d295b421f67a4d0e962b6c2365248448 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c3a12e5462468b875b5d0fbdd048de" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45149 } } peers { permanent_uuid: "b1109a56871a442584d8a397a4c72542" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38529 } } peers { permanent_uuid: "d295b421f67a4d0e962b6c2365248448" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36957 } }
I20260926 16:31:33.667788 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:33.663889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:33939 (local address 127.6.33.1:49013)
0926 16:31:33.667465 (+ 3576us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:33.667576 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:33939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3479,"thread_start_us":3331,"threads_started":1}
I20260926 16:31:33.668370 13770 ts_tablet_manager.cc:1428] T b9ba1366189c404080acbd0b54ba17eb P d295b421f67a4d0e962b6c2365248448: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260926 16:31:33.669430 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:33.668981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:33939 (local address 127.6.33.1:42621)
0926 16:31:33.669204 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:33.669268 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:33939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
I20260926 16:31:33.670725 13537 heartbeater.cc:503] Master 127.6.33.61:37487 was elected leader, sending a full tablet report...
I20260926 16:31:33.670904 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:33.668713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:33939 (local address 127.6.33.2:49155)
0926 16:31:33.670456 (+ 1743us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:33.670603 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:33939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1599,"thread_start_us":1256,"threads_started":1}
I20260926 16:31:33.671947 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:33.671552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:33939 (local address 127.6.33.2:42697)
0926 16:31:33.671738 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:33.671811 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:33939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260926 16:31:33.896080 13538 tablet.cc:2369] T b9ba1366189c404080acbd0b54ba17eb P d295b421f67a4d0e962b6c2365248448: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:31:33.971659 13779 raft_consensus.cc:482] T b9ba1366189c404080acbd0b54ba17eb P b1109a56871a442584d8a397a4c72542 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:31:33.971943 13779 raft_consensus.cc:504] T b9ba1366189c404080acbd0b54ba17eb P b1109a56871a442584d8a397a4c72542 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c3a12e5462468b875b5d0fbdd048de" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45149 } } peers { permanent_uuid: "b1109a56871a442584d8a397a4c72542" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38529 } } peers { permanent_uuid: "d295b421f67a4d0e962b6c2365248448" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36957 } }
I20260926 16:31:33.973578 13779 leader_election.cc:288] T b9ba1366189c404080acbd0b54ba17eb P b1109a56871a442584d8a397a4c72542 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93c3a12e5462468b875b5d0fbdd048de (127.6.33.3:45149), d295b421f67a4d0e962b6c2365248448 (127.6.33.2:36957)
I20260926 16:31:33.980419 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:33.974215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:39515 (local address 127.6.33.3:45149)
0926 16:31:33.974444 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:33.974450 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:33.974473 (+ 23us) server_negotiation.cc:407] Connection header received
0926 16:31:33.974595 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:33.974602 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:33.974694 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:33.974795 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:31:33.975271 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:33.976460 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:33.978339 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:33.979376 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:33.979473 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:33.979548 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:33.979949 (+ 401us) server_negotiation.cc:299] Negotiation successful
0926 16:31:33.980133 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260926 16:31:33.980456 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:33.973977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.3:45149 (local address 127.6.33.1:39515)
0926 16:31:33.974209 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:33.974231 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:33.974455 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:33.974836 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:33.974844 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:33.974881 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:31:33.975149 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:33.975157 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:33.976587 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:33.976593 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:33.978165 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:33.978177 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:33.979554 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:33.979562 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:33.979753 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:33.979801 (+ 48us) client_negotiation.cc:769] Sending connection context
0926 16:31:33.979910 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 16:31:33.980097 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260926 16:31:33.981348 13769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:33.974434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:55143 (local address 127.6.33.2:36957)
0926 16:31:33.974665 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:33.974674 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:33.975146 (+ 472us) server_negotiation.cc:407] Connection header received
0926 16:31:33.975352 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:33.975356 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:33.975438 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:33.975505 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:31:33.976129 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:33.977036 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:33.978992 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:33.980184 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:33.980323 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:33.980415 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:33.980953 (+ 538us) server_negotiation.cc:299] Negotiation successful
0926 16:31:33.981051 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 16:31:33.981315 13631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9ba1366189c404080acbd0b54ba17eb" candidate_uuid: "b1109a56871a442584d8a397a4c72542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93c3a12e5462468b875b5d0fbdd048de" is_pre_election: true
I20260926 16:31:33.981437 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:33.974316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.2:36957 (local address 127.6.33.1:55143)
0926 16:31:33.975048 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:33.975061 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:33.975199 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:33.975606 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:33.975611 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:33.975638 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:31:33.975979 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:33.975990 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:33.977172 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:33.977176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:33.978821 (+ 1645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:33.978835 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:33.980474 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:33.980483 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:33.980708 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:33.980732 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 16:31:33.980905 (+ 173us) client_negotiation.cc:240] Negotiation successful
0926 16:31:33.981031 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":361,"threads_started":1}
I20260926 16:31:33.981587 13631 raft_consensus.cc:2454] T b9ba1366189c404080acbd0b54ba17eb P 93c3a12e5462468b875b5d0fbdd048de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1109a56871a442584d8a397a4c72542 in term 0.
I20260926 16:31:33.982190 13277 leader_election.cc:302] T b9ba1366189c404080acbd0b54ba17eb P b1109a56871a442584d8a397a4c72542 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93c3a12e5462468b875b5d0fbdd048de, b1109a56871a442584d8a397a4c72542; no voters:
I20260926 16:31:33.982390 13490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9ba1366189c404080acbd0b54ba17eb" candidate_uuid: "b1109a56871a442584d8a397a4c72542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d295b421f67a4d0e962b6c2365248448" is_pre_election: true
I20260926 16:31:33.982625 13779 raft_consensus.cc:2781] T b9ba1366189c404080acbd0b54ba17eb P b1109a56871a442584d8a397a4c72542 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:31:33.982753 13779 raft_consensus.cc:482] T b9ba1366189c404080acbd0b54ba17eb P b1109a56871a442584d8a397a4c72542 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:31:33.982895 13779 raft_consensus.cc:3037] T b9ba1366189c404080acbd0b54ba17eb P b1109a56871a442584d8a397a4c72542 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:31:33.982829 13490 raft_consensus.cc:2454] T b9ba1366189c404080acbd0b54ba17eb P d295b421f67a4d0e962b6c2365248448 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1109a56871a442584d8a397a4c72542 in term 0.
I20260926 16:31:33.985356 13779 raft_consensus.cc:504] T b9ba1366189c404080acbd0b54ba17eb P b1109a56871a442584d8a397a4c72542 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c3a12e5462468b875b5d0fbdd048de" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45149 } } peers { permanent_uuid: "b1109a56871a442584d8a397a4c72542" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38529 } } peers { permanent_uuid: "d295b421f67a4d0e962b6c2365248448" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36957 } }
I20260926 16:31:33.985949 13779 leader_election.cc:288] T b9ba1366189c404080acbd0b54ba17eb P b1109a56871a442584d8a397a4c72542 [CANDIDATE]: Term 1 election: Requested vote from peers 93c3a12e5462468b875b5d0fbdd048de (127.6.33.3:45149), d295b421f67a4d0e962b6c2365248448 (127.6.33.2:36957)
I20260926 16:31:33.986459 13631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9ba1366189c404080acbd0b54ba17eb" candidate_uuid: "b1109a56871a442584d8a397a4c72542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93c3a12e5462468b875b5d0fbdd048de"
I20260926 16:31:33.986531 13490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9ba1366189c404080acbd0b54ba17eb" candidate_uuid: "b1109a56871a442584d8a397a4c72542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d295b421f67a4d0e962b6c2365248448"
I20260926 16:31:33.986691 13631 raft_consensus.cc:3037] T b9ba1366189c404080acbd0b54ba17eb P 93c3a12e5462468b875b5d0fbdd048de [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:31:33.986701 13490 raft_consensus.cc:3037] T b9ba1366189c404080acbd0b54ba17eb P d295b421f67a4d0e962b6c2365248448 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:31:33.989778 13631 raft_consensus.cc:2454] T b9ba1366189c404080acbd0b54ba17eb P 93c3a12e5462468b875b5d0fbdd048de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1109a56871a442584d8a397a4c72542 in term 1.
I20260926 16:31:33.989777 13490 raft_consensus.cc:2454] T b9ba1366189c404080acbd0b54ba17eb P d295b421f67a4d0e962b6c2365248448 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1109a56871a442584d8a397a4c72542 in term 1.
I20260926 16:31:33.990216 13277 leader_election.cc:302] T b9ba1366189c404080acbd0b54ba17eb P b1109a56871a442584d8a397a4c72542 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b1109a56871a442584d8a397a4c72542, d295b421f67a4d0e962b6c2365248448; no voters:
I20260926 16:31:33.990499 13779 raft_consensus.cc:2781] T b9ba1366189c404080acbd0b54ba17eb P b1109a56871a442584d8a397a4c72542 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:31:33.990942 13779 raft_consensus.cc:686] T b9ba1366189c404080acbd0b54ba17eb P b1109a56871a442584d8a397a4c72542 [term 1 LEADER]: Becoming Leader. State: Replica: b1109a56871a442584d8a397a4c72542, State: Running, Role: LEADER
I20260926 16:31:33.991520 13779 consensus_queue.cc:237] T b9ba1366189c404080acbd0b54ba17eb P b1109a56871a442584d8a397a4c72542 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c3a12e5462468b875b5d0fbdd048de" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45149 } } peers { permanent_uuid: "b1109a56871a442584d8a397a4c72542" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38529 } } peers { permanent_uuid: "d295b421f67a4d0e962b6c2365248448" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36957 } }
I20260926 16:31:33.991997 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:33.991525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:33939 (local address 127.6.33.1:37039)
0926 16:31:33.991708 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:33.991841 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:33939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 16:31:33.995283 13020 catalog_manager.cc:5495] T b9ba1366189c404080acbd0b54ba17eb P b1109a56871a442584d8a397a4c72542 reported cstate change: term changed from 0 to 1, leader changed from <none> to b1109a56871a442584d8a397a4c72542 (127.6.33.1). New cstate: current_term: 1 leader_uuid: "b1109a56871a442584d8a397a4c72542" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c3a12e5462468b875b5d0fbdd048de" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1109a56871a442584d8a397a4c72542" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d295b421f67a4d0e962b6c2365248448" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36957 } health_report { overall_health: UNKNOWN } } }
I20260926 16:31:34.079234 13748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:34.078766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:33939 (local address 127.0.0.1:50650)
0926 16:31:34.078990 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:34.079097 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:33939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260926 16:31:34.080366 13748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:34.079981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:33939 (local address 127.0.0.1:50666)
0926 16:31:34.080120 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:34.080187 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:33939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 16:31:34.097077 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:34.091493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39922 (local address 127.6.33.1:38529)
0926 16:31:34.091751 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:34.091760 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:34.092046 (+ 286us) server_negotiation.cc:407] Connection header received
0926 16:31:34.092285 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:34.092291 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:34.092407 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:34.092570 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:31:34.092987 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:34.093678 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:34.094925 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:34.095382 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:34.095506 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:34.095916 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:31:34.096494 (+ 578us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:31:34.096642 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:34.096697 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 16:31:34.096802 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 16:31:34.102025 13631 raft_consensus.cc:1261] T b9ba1366189c404080acbd0b54ba17eb P 93c3a12e5462468b875b5d0fbdd048de [term 1 FOLLOWER]: Refusing update from remote peer b1109a56871a442584d8a397a4c72542: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:31:34.102353 13490 raft_consensus.cc:1261] T b9ba1366189c404080acbd0b54ba17eb P d295b421f67a4d0e962b6c2365248448 [term 1 FOLLOWER]: Refusing update from remote peer b1109a56871a442584d8a397a4c72542: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:31:34.103112 13779 consensus_queue.cc:1035] T b9ba1366189c404080acbd0b54ba17eb P b1109a56871a442584d8a397a4c72542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93c3a12e5462468b875b5d0fbdd048de" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:31:34.103492 13784 consensus_queue.cc:1035] T b9ba1366189c404080acbd0b54ba17eb P b1109a56871a442584d8a397a4c72542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d295b421f67a4d0e962b6c2365248448" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:31:34.106963 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:34.106206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:33939 (local address 127.6.33.3:36507)
0926 16:31:34.106580 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:34.106706 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:33939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
I20260926 16:31:34.115190 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:34.114648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:33939 (local address 127.6.33.2:56371)
0926 16:31:34.114879 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:34.114992 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:33939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20260926 16:31:34.117095 13788 mvcc.cc:205] Tried to move back new op lower bound from 7333643444632809472 to 7333643444193710080. Current Snapshot: MvccSnapshot[applied={T|T < 7333643444632809472}]
I20260926 16:31:34.118830 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:34.118267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:33939 (local address 127.6.33.3:36091)
0926 16:31:34.118533 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:34.118649 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:33939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
I20260926 16:31:34.121765 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:34.121177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:33939 (local address 127.6.33.1:38589)
0926 16:31:34.121487 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:34.121573 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:33939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192}
I20260926 16:31:34.135048 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:34.134590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:33939 (local address 127.6.33.1:53185)
0926 16:31:34.134761 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:34.134854 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:33939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 16:31:34.134995 13020 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:31:34.135340 13020 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:31:34.140362 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:34.139881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:33939 (local address 127.6.33.2:35071)
0926 16:31:34.140096 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:34.140190 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:33939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260926 16:31:34.145328 13020 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9d1dfe3b814a4549aebd349a75850ef4: Sending DeleteTablet for 3 replicas of tablet b9ba1366189c404080acbd0b54ba17eb
I20260926 16:31:34.146524 13324 tablet_service.cc:1514] Processing DeleteTablet for tablet b9ba1366189c404080acbd0b54ba17eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:31:34 UTC) from {username='oryx', principal='oryx/127.6.33.61@KRBTEST.COM'} at 127.0.0.1:39920
I20260926 16:31:34.147229 13611 tablet_service.cc:1514] Processing DeleteTablet for tablet b9ba1366189c404080acbd0b54ba17eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:31:34 UTC) from {username='oryx', principal='oryx/127.6.33.61@KRBTEST.COM'} at 127.0.0.1:58636
I20260926 16:31:34.147289 13798 tablet_replica.cc:330] T b9ba1366189c404080acbd0b54ba17eb P b1109a56871a442584d8a397a4c72542: stopping tablet replica
I20260926 16:31:34.147746 13798 raft_consensus.cc:2229] T b9ba1366189c404080acbd0b54ba17eb P b1109a56871a442584d8a397a4c72542 [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:31:34.147897 13799 tablet_replica.cc:330] T b9ba1366189c404080acbd0b54ba17eb P 93c3a12e5462468b875b5d0fbdd048de: stopping tablet replica
I20260926 16:31:34.148078 13799 raft_consensus.cc:2229] T b9ba1366189c404080acbd0b54ba17eb P 93c3a12e5462468b875b5d0fbdd048de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:31:34.148372 13799 raft_consensus.cc:2258] T b9ba1366189c404080acbd0b54ba17eb P 93c3a12e5462468b875b5d0fbdd048de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:31:34.148547 13470 tablet_service.cc:1514] Processing DeleteTablet for tablet b9ba1366189c404080acbd0b54ba17eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:31:34 UTC) from {username='oryx', principal='oryx/127.6.33.61@KRBTEST.COM'} at 127.0.0.1:40200
I20260926 16:31:34.148638 13469 tablet_service.cc:1514] Processing DeleteTablet for tablet b9ba1366189c404080acbd0b54ba17eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:31:34 UTC) from {username='oryx', principal='oryx/127.6.33.61@KRBTEST.COM'} at 127.0.0.1:40200
I20260926 16:31:34.149195 13800 tablet_replica.cc:330] T b9ba1366189c404080acbd0b54ba17eb P d295b421f67a4d0e962b6c2365248448: stopping tablet replica
I20260926 16:31:34.149385 13800 raft_consensus.cc:2229] T b9ba1366189c404080acbd0b54ba17eb P d295b421f67a4d0e962b6c2365248448 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:31:34.149649 13800 raft_consensus.cc:2258] T b9ba1366189c404080acbd0b54ba17eb P d295b421f67a4d0e962b6c2365248448 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:31:34.149724 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 13186
I20260926 16:31:34.150084 13799 ts_tablet_manager.cc:1905] T b9ba1366189c404080acbd0b54ba17eb P 93c3a12e5462468b875b5d0fbdd048de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:31:34.153543 13800 ts_tablet_manager.cc:1905] T b9ba1366189c404080acbd0b54ba17eb P d295b421f67a4d0e962b6c2365248448: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:31:34.157073 13798 raft_consensus.cc:2258] T b9ba1366189c404080acbd0b54ba17eb P b1109a56871a442584d8a397a4c72542 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:31:34.158593 13798 ts_tablet_manager.cc:1905] T b9ba1366189c404080acbd0b54ba17eb P b1109a56871a442584d8a397a4c72542: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:31:34.160398 13800 ts_tablet_manager.cc:1918] T b9ba1366189c404080acbd0b54ba17eb P d295b421f67a4d0e962b6c2365248448: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:31:34.160540 13800 log.cc:1196] T b9ba1366189c404080acbd0b54ba17eb P d295b421f67a4d0e962b6c2365248448: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-1/wal/wals/b9ba1366189c404080acbd0b54ba17eb
I20260926 16:31:34.161036 13800 ts_tablet_manager.cc:1939] T b9ba1366189c404080acbd0b54ba17eb P d295b421f67a4d0e962b6c2365248448: Deleting consensus metadata
I20260926 16:31:34.162292 13007 catalog_manager.cc:4841] TS d295b421f67a4d0e962b6c2365248448 (127.6.33.2:36957): tablet b9ba1366189c404080acbd0b54ba17eb (table test-table [id=468031f9ed074bc984efcaac97283326]) successfully deleted
I20260926 16:31:34.162999 13799 ts_tablet_manager.cc:1918] T b9ba1366189c404080acbd0b54ba17eb P 93c3a12e5462468b875b5d0fbdd048de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:31:34.163213 13799 log.cc:1196] T b9ba1366189c404080acbd0b54ba17eb P 93c3a12e5462468b875b5d0fbdd048de: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-2/wal/wals/b9ba1366189c404080acbd0b54ba17eb
W20260926 16:31:34.163482 13007 catalog_manager.cc:4808] TS d295b421f67a4d0e962b6c2365248448 (127.6.33.2:36957): delete failed for tablet b9ba1366189c404080acbd0b54ba17eb because the tablet was not found. No further retry: Not found: Tablet not found: b9ba1366189c404080acbd0b54ba17eb
I20260926 16:31:34.163828 13799 ts_tablet_manager.cc:1939] T b9ba1366189c404080acbd0b54ba17eb P 93c3a12e5462468b875b5d0fbdd048de: Deleting consensus metadata
I20260926 16:31:34.164933 13007 catalog_manager.cc:4841] TS 93c3a12e5462468b875b5d0fbdd048de (127.6.33.3:45149): tablet b9ba1366189c404080acbd0b54ba17eb (table test-table [id=468031f9ed074bc984efcaac97283326]) successfully deleted
I20260926 16:31:34.171439 13798 ts_tablet_manager.cc:1918] T b9ba1366189c404080acbd0b54ba17eb P b1109a56871a442584d8a397a4c72542: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:31:34.171619 13798 log.cc:1196] T b9ba1366189c404080acbd0b54ba17eb P b1109a56871a442584d8a397a4c72542: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440233850748-6276-0/minicluster-data/ts-0/wal/wals/b9ba1366189c404080acbd0b54ba17eb
I20260926 16:31:34.172096 13798 ts_tablet_manager.cc:1939] T b9ba1366189c404080acbd0b54ba17eb P b1109a56871a442584d8a397a4c72542: Deleting consensus metadata
I20260926 16:31:34.173321 13007 catalog_manager.cc:4841] TS b1109a56871a442584d8a397a4c72542 (127.6.33.1:38529): tablet b9ba1366189c404080acbd0b54ba17eb (table test-table [id=468031f9ed074bc984efcaac97283326]) successfully deleted
I20260926 16:31:34.176646 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:34.151401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39924 (local address 127.6.33.1:38529)
0926 16:31:34.151787 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:34.151794 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:34.161860 (+ 10066us) server_negotiation.cc:407] Connection header received
0926 16:31:34.162046 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:34.162051 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:34.162144 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:34.162254 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:34.164191 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:34.165342 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:34.166765 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:34.167258 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:34.167412 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:34.170246 (+ 2834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:34.170265 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:34.170280 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:34.170352 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:34.173897 (+ 3545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:34.174549 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:34.174557 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:34.174559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:34.174698 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:34.175183 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:34.175187 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:34.175188 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:34.175524 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:34.175752 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:34.176133 (+ 381us) server_negotiation.cc:299] Negotiation successful
0926 16:31:34.176297 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227}
I20260926 16:31:34.177345 13384 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:31:34.286774 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 13186
I20260926 16:31:34.305739 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 13401
I20260926 16:31:34.318641 13769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:34.306495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40210 (local address 127.6.33.2:36957)
0926 16:31:34.306666 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:34.306670 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:34.306690 (+ 20us) server_negotiation.cc:407] Connection header received
0926 16:31:34.306849 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:34.306853 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:34.306940 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:34.307027 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:34.308204 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:34.309010 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:34.310204 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:34.310598 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:34.310693 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:34.313399 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:34.313413 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:34.313429 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:34.313460 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:34.316277 (+ 2817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:34.316953 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:34.316960 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:34.316963 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:34.317080 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:34.317386 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:34.317390 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:34.317393 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:34.317737 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:34.317972 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:34.318199 (+ 227us) server_negotiation.cc:299] Negotiation successful
0926 16:31:34.318368 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 16:31:34.319303 13530 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:31:34.437114 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 13401
I20260926 16:31:34.458591 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 13548
I20260926 16:31:34.472337 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:34.459432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58648 (local address 127.6.33.3:45149)
0926 16:31:34.459608 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:34.459616 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:34.459687 (+ 71us) server_negotiation.cc:407] Connection header received
0926 16:31:34.459900 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:34.459906 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:34.459980 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:34.460081 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:34.461478 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:34.462178 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:34.463357 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:34.463757 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:34.463887 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:34.466616 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:34.466632 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:34.466648 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:34.466681 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:34.469597 (+ 2916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:34.470289 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:34.470305 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:34.470307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:34.470425 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:34.470790 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:34.470797 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:34.470800 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:34.471168 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:34.471470 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:34.471779 (+ 309us) server_negotiation.cc:299] Negotiation successful
0926 16:31:34.472007 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 16:31:34.473078 13671 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:31:34.567193 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 13548
I20260926 16:31:34.584146 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 12884
I20260926 16:31:34.585345 13050 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:31:34.706159 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 12884
W20260926 16:31:34.727074 13193 connection.cc:535] server connection from 127.0.0.1:46254 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 16:31:34.727392 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 12984
I20260926 16:31:34.728435 13239 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:31:34.842276 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 12984
2026-09-26T16:31:34Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9811 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:31:34 dist-test-slave-khd3 krb5kdc[13810](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:31:34 dist-test-slave-khd3 krb5kdc[13810](info): set up 2 sockets
Sep 26 16:31:34 dist-test-slave-khd3 krb5kdc[13810](info): commencing operation
krb5kdc: starting...
W20260926 16:31:35.946820 6276 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Sep 26 16:31:36 dist-test-slave-khd3 krb5kdc[13810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:31:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:31:36Z Disabled control of system clock
WARNING: no policy specified for oryx/127.6.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/krb5kdc/oryx_127.6.33.62.keytab.
Entry for principal oryx/127.6.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/krb5kdc/oryx_127.6.33.62.keytab.
I20260926 16:31:36.173461 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:38215
--webserver_interface=127.6.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/krb5kdc/oryx_127.6.33.62.keytab
--principal=oryx/127.6.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:36601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.33.62:38215
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:31:36.498085 13823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:31:36.498442 13823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:31:36.498595 13823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:31:36.505832 13823 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:31:36.505970 13823 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 16:31:36.506068 13823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:31:36.506177 13823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:31:36.506259 13823 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:31:36.506357 13823 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:31:36.516057 13823 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:36601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.33.62:38215
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/krb5kdc/oryx_127.6.33.62.keytab
--principal=oryx/127.6.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:38215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:31:36.517448 13823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:31:36.518852 13823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:31:36.530637 13828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:36.530675 13829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:36.530709 13831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:31:36.531517 13823 server_base.cc:1034] running on GCE node
I20260926 16:31:36.532302 13823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:31:36.533844 13823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:31:36.535096 13823 hybrid_clock.cc:648] HybridClock initialized: now 1790440296535033 us; error 44 us; skew 500 ppm
I20260926 16:31:36.535436 13823 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:31:36 dist-test-slave-khd3 krb5kdc[13810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440296, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:31:36.538724 13823 init.cc:377] Logged in from keytab as oryx/127.6.33.62@KRBTEST.COM (short username oryx)
I20260926 16:31:36.540128 13823 webserver.cc:460] Webserver started at http://127.6.33.62:39807/ using document root <none> and password file <none>
I20260926 16:31:36.540856 13823 fs_manager.cc:359] Metadata directory not provided
I20260926 16:31:36.540954 13823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:31:36.541221 13823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:31:36.544256 13823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/master-0/data/instance:
uuid: "f9211160536747d4bb0d0bceaaae9ffb"
format_stamp: "Formatted at 2026-09-26 16:31:36 on dist-test-slave-khd3"
server_key: "63708fb93c8e20543b6955dbe6677255"
server_key_iv: "b13508da00716958a3d64dfd918e04b3"
server_key_version: "encryptionkey@0"
I20260926 16:31:36.545104 13823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/master-0/wal/instance:
uuid: "f9211160536747d4bb0d0bceaaae9ffb"
format_stamp: "Formatted at 2026-09-26 16:31:36 on dist-test-slave-khd3"
server_key: "63708fb93c8e20543b6955dbe6677255"
server_key_iv: "b13508da00716958a3d64dfd918e04b3"
server_key_version: "encryptionkey@0"
I20260926 16:31:36.550302 13823 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 16:31:36.553747 13838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:36.555126 13823 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:31:36.555261 13823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/master-0/wal
uuid: "f9211160536747d4bb0d0bceaaae9ffb"
format_stamp: "Formatted at 2026-09-26 16:31:36 on dist-test-slave-khd3"
server_key: "63708fb93c8e20543b6955dbe6677255"
server_key_iv: "b13508da00716958a3d64dfd918e04b3"
server_key_version: "encryptionkey@0"
I20260926 16:31:36.555438 13823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:31:36.580057 13823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:31:36.584558 13823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:31:36.584830 13823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:31:36.606724 13823 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.62:38215
I20260926 16:31:36.606745 13899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.62:38215 every 8 connection(s)
I20260926 16:31:36.608094 13823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
I20260926 16:31:36.611824 13900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:31:36.614810 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 13823
I20260926 16:31:36.615054 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/master-0/wal/instance
I20260926 16:31:36.615818 6276 external_mini_cluster.cc:1428] Setting key 495aa59316a40a7e11437ff1cc4d587f
I20260926 16:31:36.623860 13900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9211160536747d4bb0d0bceaaae9ffb: Bootstrap starting.
I20260926 16:31:36.626822 13900 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9211160536747d4bb0d0bceaaae9ffb: Neither blocks nor log segments found. Creating new log.
I20260926 16:31:36.628154 13900 log.cc:824] T 00000000000000000000000000000000 P f9211160536747d4bb0d0bceaaae9ffb: Log is configured to *not* fsync() on all Append() calls
Sep 26 16:31:36 dist-test-slave-khd3 krb5kdc[13810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM
I20260926 16:31:36.631314 13900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9211160536747d4bb0d0bceaaae9ffb: No bootstrap required, opened a new log
I20260926 16:31:36.638257 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:36.618030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44812 (local address 127.6.33.62:38215)
0926 16:31:36.618783 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:36.618795 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:36.618834 (+ 39us) server_negotiation.cc:407] Connection header received
0926 16:31:36.620492 (+ 1658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:36.620514 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:36.620993 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:36.621401 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:36.622815 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:36.624151 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:36.625436 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:36.626092 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:36.626288 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:36.631085 (+ 4797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:36.631092 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:36.631106 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:36.631138 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:36.634334 (+ 3196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:36.635099 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:36.635105 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:36.635111 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:36.635265 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:36.635651 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:36.635657 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:36.635659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:36.636155 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:36.636378 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:36.636841 (+ 463us) server_negotiation.cc:299] Negotiation successful
0926 16:31:36.637266 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":356,"threads_started":1}
I20260926 16:31:36.639950 13900 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9211160536747d4bb0d0bceaaae9ffb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9211160536747d4bb0d0bceaaae9ffb" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 38215 } }
I20260926 16:31:36.640282 13900 raft_consensus.cc:374] T 00000000000000000000000000000000 P f9211160536747d4bb0d0bceaaae9ffb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:31:36.640407 13900 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9211160536747d4bb0d0bceaaae9ffb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9211160536747d4bb0d0bceaaae9ffb, State: Initialized, Role: FOLLOWER
I20260926 16:31:36.641031 13900 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9211160536747d4bb0d0bceaaae9ffb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9211160536747d4bb0d0bceaaae9ffb" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 38215 } }
I20260926 16:31:36.641259 13900 raft_consensus.cc:388] T 00000000000000000000000000000000 P f9211160536747d4bb0d0bceaaae9ffb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:31:36.641417 13900 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9211160536747d4bb0d0bceaaae9ffb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:31:36.641593 13900 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9211160536747d4bb0d0bceaaae9ffb [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:31:36.643615 13900 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9211160536747d4bb0d0bceaaae9ffb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9211160536747d4bb0d0bceaaae9ffb" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 38215 } }
I20260926 16:31:36.644096 13900 leader_election.cc:302] T 00000000000000000000000000000000 P f9211160536747d4bb0d0bceaaae9ffb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f9211160536747d4bb0d0bceaaae9ffb; no voters:
I20260926 16:31:36.644629 13900 leader_election.cc:288] T 00000000000000000000000000000000 P f9211160536747d4bb0d0bceaaae9ffb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:31:36.644758 13905 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9211160536747d4bb0d0bceaaae9ffb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:31:36.645453 13905 raft_consensus.cc:686] T 00000000000000000000000000000000 P f9211160536747d4bb0d0bceaaae9ffb [term 1 LEADER]: Becoming Leader. State: Replica: f9211160536747d4bb0d0bceaaae9ffb, State: Running, Role: LEADER
I20260926 16:31:36.645946 13905 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9211160536747d4bb0d0bceaaae9ffb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9211160536747d4bb0d0bceaaae9ffb" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 38215 } }
I20260926 16:31:36.646632 13900 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9211160536747d4bb0d0bceaaae9ffb [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:31:36.648671 13907 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9211160536747d4bb0d0bceaaae9ffb [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9211160536747d4bb0d0bceaaae9ffb. Latest consensus state: current_term: 1 leader_uuid: "f9211160536747d4bb0d0bceaaae9ffb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9211160536747d4bb0d0bceaaae9ffb" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 38215 } } }
I20260926 16:31:36.649042 13907 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9211160536747d4bb0d0bceaaae9ffb [sys.catalog]: This master's current role is: LEADER
I20260926 16:31:36.649991 13906 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9211160536747d4bb0d0bceaaae9ffb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f9211160536747d4bb0d0bceaaae9ffb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9211160536747d4bb0d0bceaaae9ffb" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 38215 } } }
I20260926 16:31:36.650327 13906 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9211160536747d4bb0d0bceaaae9ffb [sys.catalog]: This master's current role is: LEADER
I20260926 16:31:36.652994 13912 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:31:36.658607 13912 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:31:36.666792 13912 catalog_manager.cc:1347] Generated new cluster ID: dda32797c994406bbe3e4f0261730415
I20260926 16:31:36.666920 13912 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:31:36.681073 13912 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:31:36.682447 13912 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:31:36.693087 13912 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f9211160536747d4bb0d0bceaaae9ffb: Generated new TSK 0
I20260926 16:31:36.693974 13912 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.6.33.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.33.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.33.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/krb5kdc/oryx_127.6.33.1.keytab.
Entry for principal oryx/127.6.33.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/krb5kdc/oryx_127.6.33.1.keytab.
I20260926 16:31:36.753403 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.1:0
--local_ip_for_outbound_sockets=127.6.33.1
--webserver_interface=127.6.33.1
--webserver_port=0
--tserver_master_addrs=127.6.33.62:38215
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/krb5kdc/oryx_127.6.33.1.keytab
--principal=oryx/127.6.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:36601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:31:37.090137 13926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:31:37.090425 13926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:31:37.090590 13926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:31:37.097901 13926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:31:37.098013 13926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:31:37.098155 13926 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 16:31:37.098286 13926 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.1
I20260926 16:31:37.108156 13926 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:36601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/krb5kdc/oryx_127.6.33.1.keytab
--principal=oryx/127.6.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.33.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.33.62:38215
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.1
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:31:37.109512 13926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:31:37.110821 13926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:31:37.124402 13931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:37.124436 13932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:37.124402 13934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:31:37.125689 13926 server_base.cc:1034] running on GCE node
I20260926 16:31:37.126268 13926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:31:37.127202 13926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:31:37.128428 13926 hybrid_clock.cc:648] HybridClock initialized: now 1790440297128358 us; error 35 us; skew 500 ppm
I20260926 16:31:37.128738 13926 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:31:37 dist-test-slave-khd3 krb5kdc[13810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440297, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:31:37.132092 13926 init.cc:377] Logged in from keytab as oryx/127.6.33.1@KRBTEST.COM (short username oryx)
I20260926 16:31:37.133574 13926 webserver.cc:460] Webserver started at http://127.6.33.1:40243/ using document root <none> and password file <none>
I20260926 16:31:37.134243 13926 fs_manager.cc:359] Metadata directory not provided
I20260926 16:31:37.134328 13926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:31:37.134588 13926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:31:37.137472 13926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/ts-0/data/instance:
uuid: "ee5950d438ab4f53bab4f057a1712254"
format_stamp: "Formatted at 2026-09-26 16:31:37 on dist-test-slave-khd3"
server_key: "fac22534effaf755dc68885ce0ca544b"
server_key_iv: "f5b65c942e7fff7f0cb6635c90dd23d6"
server_key_version: "encryptionkey@0"
I20260926 16:31:37.138187 13926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/ts-0/wal/instance:
uuid: "ee5950d438ab4f53bab4f057a1712254"
format_stamp: "Formatted at 2026-09-26 16:31:37 on dist-test-slave-khd3"
server_key: "fac22534effaf755dc68885ce0ca544b"
server_key_iv: "f5b65c942e7fff7f0cb6635c90dd23d6"
server_key_version: "encryptionkey@0"
I20260926 16:31:37.143201 13926 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 16:31:37.146554 13941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:37.148221 13926 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 16:31:37.148399 13926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/ts-0/wal
uuid: "ee5950d438ab4f53bab4f057a1712254"
format_stamp: "Formatted at 2026-09-26 16:31:37 on dist-test-slave-khd3"
server_key: "fac22534effaf755dc68885ce0ca544b"
server_key_iv: "f5b65c942e7fff7f0cb6635c90dd23d6"
server_key_version: "encryptionkey@0"
I20260926 16:31:37.148605 13926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:31:37.167850 13926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:31:37.172199 13926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:31:37.172478 13926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:31:37.175203 13926 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:31:37.175323 13926 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:37.175472 13926 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:31:37.175578 13926 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 26 16:31:37 dist-test-slave-khd3 krb5kdc[13810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440297, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.33.1@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM
I20260926 16:31:37.198454 13949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:37.175614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:39987 (local address 127.6.33.62:38215)
0926 16:31:37.176246 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:37.176254 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:37.177222 (+ 968us) server_negotiation.cc:407] Connection header received
0926 16:31:37.179237 (+ 2015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:37.179243 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:37.179331 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:37.179458 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:37.181576 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:37.182649 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:37.186002 (+ 3353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:37.186634 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:37.186794 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:37.191532 (+ 4738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:37.191540 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:37.191548 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:37.191599 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:37.194899 (+ 3300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:37.195712 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:37.195717 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:37.195721 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:37.195847 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:37.196293 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:37.196298 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:37.196304 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:37.196627 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:37.196833 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:37.198000 (+ 1167us) server_negotiation.cc:299] Negotiation successful
0926 16:31:37.198181 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":385,"threads_started":1}
I20260926 16:31:37.200917 13951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:37.176261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:38215 (local address 127.6.33.1:39987)
0926 16:31:37.177060 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:37.177099 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:37.178952 (+ 1853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:37.179702 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:37.179710 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:37.180328 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:37.181392 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:37.181413 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:37.182837 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:37.182845 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:37.185843 (+ 2998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:37.185858 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:37.187128 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:37.187133 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:37.187413 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:37.188169 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:37.188190 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:37.191213 (+ 3023us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:31:37.195061 (+ 3848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:37.195073 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:37.195092 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:37.195578 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:37.195958 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:37.195964 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:37.195968 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:37.196171 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:37.196828 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:31:37.196836 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:31:37.197457 (+ 621us) client_negotiation.cc:769] Sending connection context
0926 16:31:37.197902 (+ 445us) client_negotiation.cc:240] Negotiation successful
0926 16:31:37.198676 (+ 774us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":305,"threads_started":1}
I20260926 16:31:37.221984 14020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:37.207732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:38215 (local address 127.6.33.1:44671)
0926 16:31:37.208342 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:37.208357 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:37.208525 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:37.208908 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:37.208913 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:37.209660 (+ 747us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:37.210126 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:37.210141 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:37.211554 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:37.211565 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:37.213391 (+ 1826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:37.213406 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:37.214160 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:37.214166 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:37.214322 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:37.215023 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:37.215039 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:37.215757 (+ 718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:31:37.218741 (+ 2984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:37.218748 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:37.218751 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:37.219291 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:37.219767 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:37.219773 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:37.219777 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:37.219922 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:37.220548 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:31:37.220553 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:31:37.220823 (+ 270us) client_negotiation.cc:769] Sending connection context
0926 16:31:37.221006 (+ 183us) client_negotiation.cc:240] Negotiation successful
0926 16:31:37.221338 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":368,"threads_started":1}
I20260926 16:31:37.222635 13949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:37.207656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:44671 (local address 127.6.33.62:38215)
0926 16:31:37.207822 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:37.207826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:37.208465 (+ 639us) server_negotiation.cc:407] Connection header received
0926 16:31:37.208651 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:37.208658 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:37.208724 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:37.208881 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:37.210337 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:37.211397 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:37.213550 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:37.213976 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:37.214119 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:37.215931 (+ 1812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:37.215937 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:37.215942 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:37.215983 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:37.218612 (+ 2629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:37.219441 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:37.219448 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:37.219454 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:37.219639 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:37.220046 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:37.220051 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:37.220055 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:37.220445 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:37.220584 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:37.222034 (+ 1450us) server_negotiation.cc:299] Negotiation successful
0926 16:31:37.222301 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 16:31:37.232250 13926 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.1:32903
I20260926 16:31:37.232290 14061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.1:32903 every 8 connection(s)
I20260926 16:31:37.233489 13926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
I20260926 16:31:37.235440 14062 heartbeater.cc:347] Connected to a master server at 127.6.33.62:38215
I20260926 16:31:37.235620 14062 heartbeater.cc:464] Registering TS with master...
I20260926 16:31:37.236198 14062 heartbeater.cc:511] Master 127.6.33.62:38215 requested a full tablet report, sending...
I20260926 16:31:37.237324 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 13926
I20260926 16:31:37.237530 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/ts-0/wal/instance
I20260926 16:31:37.238129 6276 external_mini_cluster.cc:1428] Setting key d0e80f1ec5d0dd7ff642a276cae07e61
I20260926 16:31:37.238193 13855 ts_manager.cc:194] Registered new tserver with Master: ee5950d438ab4f53bab4f057a1712254 (127.6.33.1:32903)
I20260926 16:31:37.239802 13855 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.33.1@KRBTEST.COM'} at 127.6.33.1:39987
WARNING: no policy specified for oryx/127.6.33.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.33.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.33.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/krb5kdc/oryx_127.6.33.2.keytab.
Entry for principal oryx/127.6.33.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/krb5kdc/oryx_127.6.33.2.keytab.
I20260926 16:31:37.283612 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.2:0
--local_ip_for_outbound_sockets=127.6.33.2
--webserver_interface=127.6.33.2
--webserver_port=0
--tserver_master_addrs=127.6.33.62:38215
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/krb5kdc/oryx_127.6.33.2.keytab
--principal=oryx/127.6.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:36601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:31:37.615149 14066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:31:37.615545 14066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:31:37.615684 14066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:31:37.627171 14066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:31:37.627306 14066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:31:37.627496 14066 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 16:31:37.627617 14066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.2
I20260926 16:31:37.638700 14066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:36601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/krb5kdc/oryx_127.6.33.2.keytab
--principal=oryx/127.6.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.33.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.33.62:38215
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.2
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:31:37.639922 14066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:31:37.641294 14066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:31:37.654317 14074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:37.654388 14072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:37.654364 14071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:31:37.655447 14066 server_base.cc:1034] running on GCE node
I20260926 16:31:37.656082 14066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:31:37.657101 14066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:31:37.658336 14066 hybrid_clock.cc:648] HybridClock initialized: now 1790440297658259 us; error 59 us; skew 500 ppm
I20260926 16:31:37.658655 14066 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:31:37 dist-test-slave-khd3 krb5kdc[13810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440297, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:31:37.661993 14066 init.cc:377] Logged in from keytab as oryx/127.6.33.2@KRBTEST.COM (short username oryx)
I20260926 16:31:37.663441 14066 webserver.cc:460] Webserver started at http://127.6.33.2:46873/ using document root <none> and password file <none>
I20260926 16:31:37.664083 14066 fs_manager.cc:359] Metadata directory not provided
I20260926 16:31:37.664167 14066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:31:37.664433 14066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:31:37.667279 14066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/ts-1/data/instance:
uuid: "338f775607c644ba8be145b78f82edb7"
format_stamp: "Formatted at 2026-09-26 16:31:37 on dist-test-slave-khd3"
server_key: "e734ecdef18c7093557ca596ef1ab0df"
server_key_iv: "0c3e9c1c7c3dca37b122db43452d1c3c"
server_key_version: "encryptionkey@0"
I20260926 16:31:37.667989 14066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/ts-1/wal/instance:
uuid: "338f775607c644ba8be145b78f82edb7"
format_stamp: "Formatted at 2026-09-26 16:31:37 on dist-test-slave-khd3"
server_key: "e734ecdef18c7093557ca596ef1ab0df"
server_key_iv: "0c3e9c1c7c3dca37b122db43452d1c3c"
server_key_version: "encryptionkey@0"
I20260926 16:31:37.673023 14066 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260926 16:31:37.676304 14081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:37.677721 14066 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:31:37.677855 14066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/ts-1/wal
uuid: "338f775607c644ba8be145b78f82edb7"
format_stamp: "Formatted at 2026-09-26 16:31:37 on dist-test-slave-khd3"
server_key: "e734ecdef18c7093557ca596ef1ab0df"
server_key_iv: "0c3e9c1c7c3dca37b122db43452d1c3c"
server_key_version: "encryptionkey@0"
I20260926 16:31:37.678038 14066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:31:37.695238 14066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:31:37.699767 14066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:31:37.700006 14066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:31:37.702754 14066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:31:37.702879 14066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:37.703019 14066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:31:37.703084 14066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 16:31:37 dist-test-slave-khd3 krb5kdc[13810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440297, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.33.2@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM
I20260926 16:31:37.726357 13949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:37.703585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:38345 (local address 127.6.33.62:38215)
0926 16:31:37.703810 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:37.703816 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:37.705379 (+ 1563us) server_negotiation.cc:407] Connection header received
0926 16:31:37.707398 (+ 2019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:37.707404 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:37.707490 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:37.707612 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:37.709971 (+ 2359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:37.711063 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:37.713524 (+ 2461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:37.714021 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:37.714158 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:37.718755 (+ 4597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:37.718765 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:37.718773 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:37.718826 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:37.722795 (+ 3969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:37.724023 (+ 1228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:37.724028 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:37.724031 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:37.724148 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:37.724586 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:37.724593 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:37.724601 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:37.724941 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:37.725090 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:37.725791 (+ 701us) server_negotiation.cc:299] Negotiation successful
0926 16:31:37.726023 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 16:31:37.727512 14090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:37.704294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:38215 (local address 127.6.33.2:38345)
0926 16:31:37.705208 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:37.705247 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:37.707117 (+ 1870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:37.707901 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:37.707913 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:37.708710 (+ 797us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:37.709719 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:37.709749 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:37.711217 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:37.711222 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:37.713365 (+ 2143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:37.713377 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:37.714323 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:37.714333 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:37.714727 (+ 394us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:37.715637 (+ 910us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:37.715674 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:37.718430 (+ 2756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:31:37.723021 (+ 4591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:37.723032 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:37.723052 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:37.723825 (+ 773us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:37.724262 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:37.724267 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:37.724271 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:37.724467 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:37.725066 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:31:37.725074 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:31:37.725424 (+ 350us) client_negotiation.cc:769] Sending connection context
0926 16:31:37.725681 (+ 257us) client_negotiation.cc:240] Negotiation successful
0926 16:31:37.726192 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"thread_start_us":392,"threads_started":1}
I20260926 16:31:37.749367 13949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:37.733004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:45875 (local address 127.6.33.62:38215)
0926 16:31:37.733161 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:37.733165 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:37.734147 (+ 982us) server_negotiation.cc:407] Connection header received
0926 16:31:37.734345 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:37.734351 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:37.734424 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:37.734523 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:37.735615 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:37.736432 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:37.738455 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:37.738895 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:37.739044 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:37.741267 (+ 2223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:37.741273 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:37.741278 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:37.741314 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:37.745981 (+ 4667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:37.746907 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:37.746916 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:37.746922 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:37.747129 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:37.747655 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:37.747662 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:37.747667 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:37.748049 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:37.748217 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:37.748960 (+ 743us) server_negotiation.cc:299] Negotiation successful
0926 16:31:37.749141 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 16:31:37.749838 14143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:37.733180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:38215 (local address 127.6.33.2:45875)
0926 16:31:37.734043 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:37.734063 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:37.734232 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:37.734546 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:37.734550 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:37.735019 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:37.735322 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:37.735331 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:37.736577 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:37.736584 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:37.738156 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:37.738171 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:37.739089 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:37.739096 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:37.739285 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:37.739987 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:37.740004 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:37.741069 (+ 1065us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:31:37.746158 (+ 5089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:37.746165 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:37.746171 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:37.746750 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:37.747299 (+ 549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:37.747306 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:37.747311 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:37.747491 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:37.748180 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:31:37.748186 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:31:37.748459 (+ 273us) client_negotiation.cc:769] Sending connection context
0926 16:31:37.748671 (+ 212us) client_negotiation.cc:240] Negotiation successful
0926 16:31:37.749175 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":739,"thread_start_us":473,"threads_started":1}
I20260926 16:31:37.763828 14066 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.2:36057
I20260926 16:31:37.763856 14200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.2:36057 every 8 connection(s)
I20260926 16:31:37.765240 14066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
I20260926 16:31:37.767261 14201 heartbeater.cc:347] Connected to a master server at 127.6.33.62:38215
I20260926 16:31:37.767493 14201 heartbeater.cc:464] Registering TS with master...
I20260926 16:31:37.768155 14201 heartbeater.cc:511] Master 127.6.33.62:38215 requested a full tablet report, sending...
I20260926 16:31:37.769622 13855 ts_manager.cc:194] Registered new tserver with Master: 338f775607c644ba8be145b78f82edb7 (127.6.33.2:36057)
I20260926 16:31:37.770437 13855 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.33.2@KRBTEST.COM'} at 127.6.33.2:38345
I20260926 16:31:37.774991 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 14066
I20260926 16:31:37.775177 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/ts-1/wal/instance
I20260926 16:31:37.775648 6276 external_mini_cluster.cc:1428] Setting key cd1ec6f4dba65ab97f568fbcc5309af5
WARNING: no policy specified for oryx/127.6.33.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.33.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.33.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/krb5kdc/oryx_127.6.33.3.keytab.
Entry for principal oryx/127.6.33.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/krb5kdc/oryx_127.6.33.3.keytab.
I20260926 16:31:37.820541 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.3:0
--local_ip_for_outbound_sockets=127.6.33.3
--webserver_interface=127.6.33.3
--webserver_port=0
--tserver_master_addrs=127.6.33.62:38215
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/krb5kdc/oryx_127.6.33.3.keytab
--principal=oryx/127.6.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:36601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:31:38.141247 14205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:31:38.141641 14205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:31:38.141834 14205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:31:38.150151 14205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:31:38.150326 14205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:31:38.150475 14205 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 16:31:38.150616 14205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.3
I20260926 16:31:38.161989 14205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:36601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/krb5kdc/oryx_127.6.33.3.keytab
--principal=oryx/127.6.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.33.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.33.62:38215
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.3
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:31:38.163365 14205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:31:38.164836 14205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:31:38.179190 14210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:38.179160 14211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:38.179522 14213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:31:38.180567 14205 server_base.cc:1034] running on GCE node
I20260926 16:31:38.181493 14205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:31:38.182538 14205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:31:38.183887 14205 hybrid_clock.cc:648] HybridClock initialized: now 1790440298183802 us; error 53 us; skew 500 ppm
I20260926 16:31:38.184237 14205 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:31:38 dist-test-slave-khd3 krb5kdc[13810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440298, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:31:38.187704 14205 init.cc:377] Logged in from keytab as oryx/127.6.33.3@KRBTEST.COM (short username oryx)
I20260926 16:31:38.189416 14205 webserver.cc:460] Webserver started at http://127.6.33.3:41575/ using document root <none> and password file <none>
I20260926 16:31:38.190183 14205 fs_manager.cc:359] Metadata directory not provided
I20260926 16:31:38.190336 14205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:31:38.190691 14205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:31:38.193914 14205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/ts-2/data/instance:
uuid: "1eac546f77d24ea08f44d69d57b1cac5"
format_stamp: "Formatted at 2026-09-26 16:31:38 on dist-test-slave-khd3"
server_key: "122ea89f62f19c3ec006c0d188e02dc6"
server_key_iv: "2ccb6c5866fd9482d221409b014d4240"
server_key_version: "encryptionkey@0"
I20260926 16:31:38.194823 14205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/ts-2/wal/instance:
uuid: "1eac546f77d24ea08f44d69d57b1cac5"
format_stamp: "Formatted at 2026-09-26 16:31:38 on dist-test-slave-khd3"
server_key: "122ea89f62f19c3ec006c0d188e02dc6"
server_key_iv: "2ccb6c5866fd9482d221409b014d4240"
server_key_version: "encryptionkey@0"
I20260926 16:31:38.200886 14205 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260926 16:31:38.205016 14220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:38.206501 14205 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:31:38.206688 14205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/ts-2/wal
uuid: "1eac546f77d24ea08f44d69d57b1cac5"
format_stamp: "Formatted at 2026-09-26 16:31:38 on dist-test-slave-khd3"
server_key: "122ea89f62f19c3ec006c0d188e02dc6"
server_key_iv: "2ccb6c5866fd9482d221409b014d4240"
server_key_version: "encryptionkey@0"
I20260926 16:31:38.206946 14205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:31:38.241770 14205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:31:38.242554 14062 heartbeater.cc:503] Master 127.6.33.62:38215 was elected leader, sending a full tablet report...
I20260926 16:31:38.246223 14205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:31:38.246508 14205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:31:38.249212 14205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:31:38.249318 14205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:38.249501 14205 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:31:38.249576 14205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 16:31:38 dist-test-slave-khd3 krb5kdc[13810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440298, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.33.3@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM
I20260926 16:31:38.275211 14230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:38.251218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.3:48813 (local address 127.6.33.62:38215)
0926 16:31:38.251691 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:38.251698 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:38.251902 (+ 204us) server_negotiation.cc:407] Connection header received
0926 16:31:38.254896 (+ 2994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:38.254903 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:38.255005 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:38.255118 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:38.257783 (+ 2665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:38.259130 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:38.262449 (+ 3319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:38.263115 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:38.263256 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:38.267563 (+ 4307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:38.267568 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:38.267572 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:38.267606 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:38.271155 (+ 3549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:38.272345 (+ 1190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:38.272354 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:38.272371 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:38.272550 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:38.273308 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:38.273313 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:38.273319 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:38.273615 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:38.273784 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:38.274545 (+ 761us) server_negotiation.cc:299] Negotiation successful
0926 16:31:38.274773 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":693,"threads_started":1}
I20260926 16:31:38.276283 14229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:38.250729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:38215 (local address 127.6.33.3:48813)
0926 16:31:38.251732 (+ 1003us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:38.251786 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:38.254424 (+ 2638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:38.255651 (+ 1227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:38.255663 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:38.256550 (+ 887us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:38.257597 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:38.257619 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:38.259332 (+ 1713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:38.259341 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:38.262219 (+ 2878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:38.262242 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:38.263384 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:38.263396 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:38.263640 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:38.264391 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:38.264413 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:38.267307 (+ 2894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:31:38.271400 (+ 4093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:38.271409 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:38.271430 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:38.272204 (+ 774us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:38.272690 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:38.272695 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:38.272700 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:38.273132 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:38.273756 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:31:38.273764 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:31:38.274148 (+ 384us) client_negotiation.cc:769] Sending connection context
0926 16:31:38.274436 (+ 288us) client_negotiation.cc:240] Negotiation successful
0926 16:31:38.275005 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"spinlock_wait_cycles":21760,"thread_start_us":539,"threads_started":1}
I20260926 16:31:38.299685 14205 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.3:39807
I20260926 16:31:38.299690 14340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.3:39807 every 8 connection(s)
I20260926 16:31:38.301308 14205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
I20260926 16:31:38.303894 14341 heartbeater.cc:347] Connected to a master server at 127.6.33.62:38215
I20260926 16:31:38.303951 14305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:38.285578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:38215 (local address 127.6.33.3:47133)
0926 16:31:38.286827 (+ 1249us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:38.286865 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:38.287119 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:38.287662 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:38.287674 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:38.288148 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:38.288462 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:38.288472 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:38.290170 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:38.290176 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:38.292211 (+ 2035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:38.292231 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:38.293690 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:38.293702 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:38.293962 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:38.294871 (+ 909us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:38.294901 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:38.296537 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:31:38.300731 (+ 4194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:38.300735 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:38.300738 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:38.301173 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:38.301605 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:38.301609 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:38.301611 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:38.301730 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:38.302591 (+ 861us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:31:38.302596 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:31:38.302997 (+ 401us) client_negotiation.cc:769] Sending connection context
0926 16:31:38.303180 (+ 183us) client_negotiation.cc:240] Negotiation successful
0926 16:31:38.303487 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1035,"thread_start_us":855,"threads_started":1}
I20260926 16:31:38.303978 14230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:38.285462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.3:47133 (local address 127.6.33.62:38215)
0926 16:31:38.285664 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:38.285673 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:38.286990 (+ 1317us) server_negotiation.cc:407] Connection header received
0926 16:31:38.287340 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:38.287345 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:38.287465 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:38.287593 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:38.288632 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:38.289957 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:38.292426 (+ 2469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:38.293222 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:38.293382 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:38.296866 (+ 3484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:38.296876 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:38.296882 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:38.296922 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:38.300583 (+ 3661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:38.301300 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:38.301306 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:38.301309 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:38.301466 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:38.301867 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:38.301874 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:38.301879 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:38.302324 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:38.302615 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:38.303312 (+ 697us) server_negotiation.cc:299] Negotiation successful
0926 16:31:38.303540 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 16:31:38.304207 14341 heartbeater.cc:464] Registering TS with master...
I20260926 16:31:38.304947 14341 heartbeater.cc:511] Master 127.6.33.62:38215 requested a full tablet report, sending...
I20260926 16:31:38.306373 13855 ts_manager.cc:194] Registered new tserver with Master: 1eac546f77d24ea08f44d69d57b1cac5 (127.6.33.3:39807)
I20260926 16:31:38.307386 13855 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.33.3@KRBTEST.COM'} at 127.6.33.3:48813
I20260926 16:31:38.311113 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 14205
I20260926 16:31:38.311350 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/ts-2/wal/instance
I20260926 16:31:38.311775 6276 external_mini_cluster.cc:1428] Setting key 380482b548dbb614ea2ceafba2ca07ec
I20260926 16:31:38.314813 6276 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 16:31:38 dist-test-slave-khd3 krb5kdc[13810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440298, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 16:31:38 dist-test-slave-khd3 krb5kdc[13810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:31:38 dist-test-slave-khd3 krb5kdc[13810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 16:31:38.365438 14230 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:31:38.354877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44818 (local address 127.6.33.62:38215)
0926 16:31:38.355088 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:38.355094 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:38.355251 (+ 157us) server_negotiation.cc:407] Connection header received
0926 16:31:38.355478 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:38.355482 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:38.355545 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:38.355653 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:38.357073 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:38.357808 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:38.359064 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:38.359462 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:38.359597 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:38.365071 (+ 5474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44818: BlockingRecv error: recv got EOF from 127.0.0.1:44818 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 26 16:31:38 dist-test-slave-khd3 krb5kdc[13810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440298, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM
I20260926 16:31:38.382150 14230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:38.368664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44824 (local address 127.6.33.62:38215)
0926 16:31:38.368903 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:38.368909 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:38.369188 (+ 279us) server_negotiation.cc:407] Connection header received
0926 16:31:38.369390 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:38.369396 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:38.369489 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:38.369595 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:38.370727 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:38.371439 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:38.372714 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:38.373220 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:38.373356 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:38.376992 (+ 3636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:38.376997 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:38.377000 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:38.377035 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:38.379767 (+ 2732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:38.380438 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:38.380442 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:38.380444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:38.380558 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:38.380973 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:38.380976 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:38.380978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:38.381231 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:38.381481 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:38.381662 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 16:31:38.381846 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 16:31:38.386186 13855 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44824:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:31:38.393174 13855 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:31:38.418955 14359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:38.406084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.3:39807 (local address 127.0.0.1:60286)
0926 16:31:38.406934 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:38.406983 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:38.407157 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:38.408662 (+ 1505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:38.408674 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:38.408713 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:31:38.409240 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:38.409256 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:38.411715 (+ 2459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:38.411722 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:38.414798 (+ 3076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:38.414819 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:38.417553 (+ 2734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:38.417560 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:38.417823 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:38.417900 (+ 77us) client_negotiation.cc:769] Sending connection context
0926 16:31:38.418073 (+ 173us) client_negotiation.cc:240] Negotiation successful
0926 16:31:38.418347 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":437,"threads_started":1}
I20260926 16:31:38.420166 14361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:38.406354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60286 (local address 127.6.33.3:39807)
0926 16:31:38.407679 (+ 1325us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:38.407687 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:38.407720 (+ 33us) server_negotiation.cc:407] Connection header received
0926 16:31:38.407820 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:38.407831 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:38.408104 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:38.408305 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:31:38.409504 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:38.411533 (+ 2029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:38.415048 (+ 3515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:38.417012 (+ 1964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:38.417137 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:38.417410 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:38.419450 (+ 2040us) server_negotiation.cc:299] Negotiation successful
0926 16:31:38.419691 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1150,"thread_start_us":938,"threads_started":1}
I20260926 16:31:38.424088 14271 tablet_service.cc:1467] Processing CreateTablet for tablet ade0aa407af64d9ea312e7fcdb75bc77 (DEFAULT_TABLE table=test-table [id=585ef87554a446edaba01dfe192c1dfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:31:38.425727 14271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ade0aa407af64d9ea312e7fcdb75bc77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:31:38.426802 14363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:38.406268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:32903 (local address 127.0.0.1:44908)
0926 16:31:38.417329 (+ 11061us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:38.417352 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:38.417486 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:38.418380 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:38.418385 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:38.418411 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:31:38.418798 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:38.418814 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:38.421764 (+ 2950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:38.421771 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:38.423829 (+ 2058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:38.423840 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:38.425844 (+ 2004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:38.425850 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:38.426136 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:38.426176 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 16:31:38.426282 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 16:31:38.426420 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10890,"mutex_wait_us":62,"thread_start_us":12691,"threads_started":1}
I20260926 16:31:38.426813 14362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:38.407034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44908 (local address 127.6.33.1:32903)
0926 16:31:38.417585 (+ 10551us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:38.417598 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:38.417638 (+ 40us) server_negotiation.cc:407] Connection header received
0926 16:31:38.417779 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:38.417791 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:38.418145 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:38.418385 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:31:38.419388 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:38.421613 (+ 2225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:38.424028 (+ 2415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:38.425595 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:38.425735 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:38.425942 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:38.426352 (+ 410us) server_negotiation.cc:299] Negotiation successful
0926 16:31:38.426520 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10365,"thread_start_us":11494,"threads_started":1}
I20260926 16:31:38.428560 14360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:38.407986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.2:36057 (local address 127.0.0.1:58294)
0926 16:31:38.419168 (+ 11182us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:38.419184 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:38.419280 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:38.421462 (+ 2182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:38.421466 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:38.421493 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:31:38.421955 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:38.421968 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:38.423833 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:38.423837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:38.426081 (+ 2244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:38.426094 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:38.427673 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:38.427679 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:38.427850 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:38.427895 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 16:31:38.428011 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 16:31:38.428155 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11056,"mutex_wait_us":69,"thread_start_us":8995,"threads_started":1}
I20260926 16:31:38.429445 14364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:38.408446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58294 (local address 127.6.33.2:36057)
0926 16:31:38.420903 (+ 12457us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:38.420912 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:38.420939 (+ 27us) server_negotiation.cc:407] Connection header received
0926 16:31:38.421038 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:38.421045 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:38.421274 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:38.421484 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:31:38.422177 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:38.423640 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:38.426248 (+ 2608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:38.427463 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:38.427613 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:38.427814 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:38.428839 (+ 1025us) server_negotiation.cc:299] Negotiation successful
0926 16:31:38.429083 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12314,"thread_start_us":12169,"threads_started":1}
I20260926 16:31:38.431001 13992 tablet_service.cc:1467] Processing CreateTablet for tablet ade0aa407af64d9ea312e7fcdb75bc77 (DEFAULT_TABLE table=test-table [id=585ef87554a446edaba01dfe192c1dfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:31:38.432202 13992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ade0aa407af64d9ea312e7fcdb75bc77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:31:38.433205 14131 tablet_service.cc:1467] Processing CreateTablet for tablet ade0aa407af64d9ea312e7fcdb75bc77 (DEFAULT_TABLE table=test-table [id=585ef87554a446edaba01dfe192c1dfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:31:38.434581 14131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ade0aa407af64d9ea312e7fcdb75bc77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:31:38.442051 14365 tablet_bootstrap.cc:492] T ade0aa407af64d9ea312e7fcdb75bc77 P 1eac546f77d24ea08f44d69d57b1cac5: Bootstrap starting.
I20260926 16:31:38.445487 14366 tablet_bootstrap.cc:492] T ade0aa407af64d9ea312e7fcdb75bc77 P ee5950d438ab4f53bab4f057a1712254: Bootstrap starting.
I20260926 16:31:38.446344 14365 tablet_bootstrap.cc:654] T ade0aa407af64d9ea312e7fcdb75bc77 P 1eac546f77d24ea08f44d69d57b1cac5: Neither blocks nor log segments found. Creating new log.
I20260926 16:31:38.448177 14367 tablet_bootstrap.cc:492] T ade0aa407af64d9ea312e7fcdb75bc77 P 338f775607c644ba8be145b78f82edb7: Bootstrap starting.
I20260926 16:31:38.449087 14365 log.cc:824] T ade0aa407af64d9ea312e7fcdb75bc77 P 1eac546f77d24ea08f44d69d57b1cac5: Log is configured to *not* fsync() on all Append() calls
I20260926 16:31:38.449169 14366 tablet_bootstrap.cc:654] T ade0aa407af64d9ea312e7fcdb75bc77 P ee5950d438ab4f53bab4f057a1712254: Neither blocks nor log segments found. Creating new log.
I20260926 16:31:38.450479 14366 log.cc:824] T ade0aa407af64d9ea312e7fcdb75bc77 P ee5950d438ab4f53bab4f057a1712254: Log is configured to *not* fsync() on all Append() calls
I20260926 16:31:38.451568 14367 tablet_bootstrap.cc:654] T ade0aa407af64d9ea312e7fcdb75bc77 P 338f775607c644ba8be145b78f82edb7: Neither blocks nor log segments found. Creating new log.
I20260926 16:31:38.452518 14365 tablet_bootstrap.cc:492] T ade0aa407af64d9ea312e7fcdb75bc77 P 1eac546f77d24ea08f44d69d57b1cac5: No bootstrap required, opened a new log
I20260926 16:31:38.452867 14365 ts_tablet_manager.cc:1397] T ade0aa407af64d9ea312e7fcdb75bc77 P 1eac546f77d24ea08f44d69d57b1cac5: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260926 16:31:38.452840 14367 log.cc:824] T ade0aa407af64d9ea312e7fcdb75bc77 P 338f775607c644ba8be145b78f82edb7: Log is configured to *not* fsync() on all Append() calls
I20260926 16:31:38.453320 14366 tablet_bootstrap.cc:492] T ade0aa407af64d9ea312e7fcdb75bc77 P ee5950d438ab4f53bab4f057a1712254: No bootstrap required, opened a new log
I20260926 16:31:38.453593 14366 ts_tablet_manager.cc:1397] T ade0aa407af64d9ea312e7fcdb75bc77 P ee5950d438ab4f53bab4f057a1712254: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260926 16:31:38.456103 14367 tablet_bootstrap.cc:492] T ade0aa407af64d9ea312e7fcdb75bc77 P 338f775607c644ba8be145b78f82edb7: No bootstrap required, opened a new log
I20260926 16:31:38.456360 14367 ts_tablet_manager.cc:1397] T ade0aa407af64d9ea312e7fcdb75bc77 P 338f775607c644ba8be145b78f82edb7: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260926 16:31:38.461844 14366 raft_consensus.cc:348] T ade0aa407af64d9ea312e7fcdb75bc77 P ee5950d438ab4f53bab4f057a1712254 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eac546f77d24ea08f44d69d57b1cac5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39807 } } peers { permanent_uuid: "ee5950d438ab4f53bab4f057a1712254" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 32903 } } peers { permanent_uuid: "338f775607c644ba8be145b78f82edb7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36057 } }
I20260926 16:31:38.462267 14366 raft_consensus.cc:374] T ade0aa407af64d9ea312e7fcdb75bc77 P ee5950d438ab4f53bab4f057a1712254 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:31:38.462370 14366 raft_consensus.cc:729] T ade0aa407af64d9ea312e7fcdb75bc77 P ee5950d438ab4f53bab4f057a1712254 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee5950d438ab4f53bab4f057a1712254, State: Initialized, Role: FOLLOWER
I20260926 16:31:38.463052 14366 consensus_queue.cc:260] T ade0aa407af64d9ea312e7fcdb75bc77 P ee5950d438ab4f53bab4f057a1712254 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eac546f77d24ea08f44d69d57b1cac5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39807 } } peers { permanent_uuid: "ee5950d438ab4f53bab4f057a1712254" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 32903 } } peers { permanent_uuid: "338f775607c644ba8be145b78f82edb7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36057 } }
I20260926 16:31:38.464828 14366 ts_tablet_manager.cc:1428] T ade0aa407af64d9ea312e7fcdb75bc77 P ee5950d438ab4f53bab4f057a1712254: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260926 16:31:38.465270 14365 raft_consensus.cc:348] T ade0aa407af64d9ea312e7fcdb75bc77 P 1eac546f77d24ea08f44d69d57b1cac5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eac546f77d24ea08f44d69d57b1cac5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39807 } } peers { permanent_uuid: "ee5950d438ab4f53bab4f057a1712254" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 32903 } } peers { permanent_uuid: "338f775607c644ba8be145b78f82edb7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36057 } }
I20260926 16:31:38.465763 14365 raft_consensus.cc:374] T ade0aa407af64d9ea312e7fcdb75bc77 P 1eac546f77d24ea08f44d69d57b1cac5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:31:38.465895 14365 raft_consensus.cc:729] T ade0aa407af64d9ea312e7fcdb75bc77 P 1eac546f77d24ea08f44d69d57b1cac5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eac546f77d24ea08f44d69d57b1cac5, State: Initialized, Role: FOLLOWER
I20260926 16:31:38.466565 14365 consensus_queue.cc:260] T ade0aa407af64d9ea312e7fcdb75bc77 P 1eac546f77d24ea08f44d69d57b1cac5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eac546f77d24ea08f44d69d57b1cac5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39807 } } peers { permanent_uuid: "ee5950d438ab4f53bab4f057a1712254" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 32903 } } peers { permanent_uuid: "338f775607c644ba8be145b78f82edb7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36057 } }
I20260926 16:31:38.466857 14367 raft_consensus.cc:348] T ade0aa407af64d9ea312e7fcdb75bc77 P 338f775607c644ba8be145b78f82edb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eac546f77d24ea08f44d69d57b1cac5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39807 } } peers { permanent_uuid: "ee5950d438ab4f53bab4f057a1712254" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 32903 } } peers { permanent_uuid: "338f775607c644ba8be145b78f82edb7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36057 } }
I20260926 16:31:38.467278 14367 raft_consensus.cc:374] T ade0aa407af64d9ea312e7fcdb75bc77 P 338f775607c644ba8be145b78f82edb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:31:38.467372 14367 raft_consensus.cc:729] T ade0aa407af64d9ea312e7fcdb75bc77 P 338f775607c644ba8be145b78f82edb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 338f775607c644ba8be145b78f82edb7, State: Initialized, Role: FOLLOWER
I20260926 16:31:38.467936 14367 consensus_queue.cc:260] T ade0aa407af64d9ea312e7fcdb75bc77 P 338f775607c644ba8be145b78f82edb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eac546f77d24ea08f44d69d57b1cac5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39807 } } peers { permanent_uuid: "ee5950d438ab4f53bab4f057a1712254" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 32903 } } peers { permanent_uuid: "338f775607c644ba8be145b78f82edb7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36057 } }
I20260926 16:31:38.468775 14341 heartbeater.cc:503] Master 127.6.33.62:38215 was elected leader, sending a full tablet report...
I20260926 16:31:38.469353 14365 ts_tablet_manager.cc:1428] T ade0aa407af64d9ea312e7fcdb75bc77 P 1eac546f77d24ea08f44d69d57b1cac5: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260926 16:31:38.471722 14367 ts_tablet_manager.cc:1428] T ade0aa407af64d9ea312e7fcdb75bc77 P 338f775607c644ba8be145b78f82edb7: Time spent starting tablet: real 0.015s user 0.013s sys 0.004s
I20260926 16:31:38.471982 14201 heartbeater.cc:503] Master 127.6.33.62:38215 was elected leader, sending a full tablet report...
I20260926 16:31:38.483824 14372 raft_consensus.cc:482] T ade0aa407af64d9ea312e7fcdb75bc77 P 1eac546f77d24ea08f44d69d57b1cac5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:31:38.484114 14372 raft_consensus.cc:504] T ade0aa407af64d9ea312e7fcdb75bc77 P 1eac546f77d24ea08f44d69d57b1cac5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eac546f77d24ea08f44d69d57b1cac5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39807 } } peers { permanent_uuid: "ee5950d438ab4f53bab4f057a1712254" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 32903 } } peers { permanent_uuid: "338f775607c644ba8be145b78f82edb7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36057 } }
W20260926 16:31:38.485736 14063 tablet.cc:2369] T ade0aa407af64d9ea312e7fcdb75bc77 P ee5950d438ab4f53bab4f057a1712254: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:31:38.485829 14372 leader_election.cc:288] T ade0aa407af64d9ea312e7fcdb75bc77 P 1eac546f77d24ea08f44d69d57b1cac5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee5950d438ab4f53bab4f057a1712254 (127.6.33.1:32903), 338f775607c644ba8be145b78f82edb7 (127.6.33.2:36057)
I20260926 16:31:38.493160 14362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:38.486190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.3:37889 (local address 127.6.33.1:32903)
0926 16:31:38.486413 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:38.486421 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:38.486861 (+ 440us) server_negotiation.cc:407] Connection header received
0926 16:31:38.486949 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:38.486953 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:38.487039 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:38.487110 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:31:38.487590 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:38.488565 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:38.491089 (+ 2524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:38.492296 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:38.492445 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:38.492555 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:38.492846 (+ 291us) server_negotiation.cc:299] Negotiation successful
0926 16:31:38.492937 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 16:31:38.493237 14229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:38.486047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:32903 (local address 127.6.33.3:37889)
0926 16:31:38.486586 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:38.486605 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:38.486735 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:38.487174 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:38.487182 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:38.487217 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:31:38.487462 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:38.487471 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:38.488705 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:38.488709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:38.490924 (+ 2215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:38.490938 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:38.492456 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:38.492460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:38.492586 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:38.492645 (+ 59us) client_negotiation.cc:769] Sending connection context
0926 16:31:38.492783 (+ 138us) client_negotiation.cc:240] Negotiation successful
0926 16:31:38.492924 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"mutex_wait_us":98}
I20260926 16:31:38.493633 14374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:38.486244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.2:36057 (local address 127.6.33.3:50897)
0926 16:31:38.487140 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:38.487157 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:38.487292 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:38.487654 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:38.487656 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:38.487677 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:31:38.487935 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:38.487942 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:38.489146 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:38.489151 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:38.491158 (+ 2007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:38.491168 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:38.492777 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:38.492783 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:38.492957 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:38.492984 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 16:31:38.493083 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 16:31:38.493192 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":751,"mutex_wait_us":2,"spinlock_wait_cycles":768,"thread_start_us":560,"threads_started":1}
I20260926 16:31:38.494135 14012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ade0aa407af64d9ea312e7fcdb75bc77" candidate_uuid: "1eac546f77d24ea08f44d69d57b1cac5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee5950d438ab4f53bab4f057a1712254" is_pre_election: true
I20260926 16:31:38.494576 14364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:38.486604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.3:50897 (local address 127.6.33.2:36057)
0926 16:31:38.486783 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:38.486791 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:38.487251 (+ 460us) server_negotiation.cc:407] Connection header received
0926 16:31:38.487459 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:38.487463 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:38.487545 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:38.487631 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:31:38.488089 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:38.489024 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:38.491349 (+ 2325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:38.492533 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:38.492655 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:38.492755 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:38.493120 (+ 365us) server_negotiation.cc:299] Negotiation successful
0926 16:31:38.493617 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 16:31:38.494616 14012 raft_consensus.cc:2454] T ade0aa407af64d9ea312e7fcdb75bc77 P ee5950d438ab4f53bab4f057a1712254 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eac546f77d24ea08f44d69d57b1cac5 in term 0.
I20260926 16:31:38.495253 14223 leader_election.cc:302] T ade0aa407af64d9ea312e7fcdb75bc77 P 1eac546f77d24ea08f44d69d57b1cac5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eac546f77d24ea08f44d69d57b1cac5, ee5950d438ab4f53bab4f057a1712254; no voters:
I20260926 16:31:38.495651 14372 raft_consensus.cc:2781] T ade0aa407af64d9ea312e7fcdb75bc77 P 1eac546f77d24ea08f44d69d57b1cac5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:31:38.495554 14156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ade0aa407af64d9ea312e7fcdb75bc77" candidate_uuid: "1eac546f77d24ea08f44d69d57b1cac5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "338f775607c644ba8be145b78f82edb7" is_pre_election: true
I20260926 16:31:38.495816 14372 raft_consensus.cc:482] T ade0aa407af64d9ea312e7fcdb75bc77 P 1eac546f77d24ea08f44d69d57b1cac5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:31:38.495923 14372 raft_consensus.cc:3037] T ade0aa407af64d9ea312e7fcdb75bc77 P 1eac546f77d24ea08f44d69d57b1cac5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:31:38.496091 14156 raft_consensus.cc:2454] T ade0aa407af64d9ea312e7fcdb75bc77 P 338f775607c644ba8be145b78f82edb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eac546f77d24ea08f44d69d57b1cac5 in term 0.
I20260926 16:31:38.498458 14372 raft_consensus.cc:504] T ade0aa407af64d9ea312e7fcdb75bc77 P 1eac546f77d24ea08f44d69d57b1cac5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eac546f77d24ea08f44d69d57b1cac5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39807 } } peers { permanent_uuid: "ee5950d438ab4f53bab4f057a1712254" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 32903 } } peers { permanent_uuid: "338f775607c644ba8be145b78f82edb7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36057 } }
I20260926 16:31:38.499107 14372 leader_election.cc:288] T ade0aa407af64d9ea312e7fcdb75bc77 P 1eac546f77d24ea08f44d69d57b1cac5 [CANDIDATE]: Term 1 election: Requested vote from peers ee5950d438ab4f53bab4f057a1712254 (127.6.33.1:32903), 338f775607c644ba8be145b78f82edb7 (127.6.33.2:36057)
I20260926 16:31:38.499531 14012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ade0aa407af64d9ea312e7fcdb75bc77" candidate_uuid: "1eac546f77d24ea08f44d69d57b1cac5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee5950d438ab4f53bab4f057a1712254"
I20260926 16:31:38.499557 14156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ade0aa407af64d9ea312e7fcdb75bc77" candidate_uuid: "1eac546f77d24ea08f44d69d57b1cac5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "338f775607c644ba8be145b78f82edb7"
I20260926 16:31:38.499725 14012 raft_consensus.cc:3037] T ade0aa407af64d9ea312e7fcdb75bc77 P ee5950d438ab4f53bab4f057a1712254 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:31:38.499738 14156 raft_consensus.cc:3037] T ade0aa407af64d9ea312e7fcdb75bc77 P 338f775607c644ba8be145b78f82edb7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:31:38.501993 14012 raft_consensus.cc:2454] T ade0aa407af64d9ea312e7fcdb75bc77 P ee5950d438ab4f53bab4f057a1712254 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eac546f77d24ea08f44d69d57b1cac5 in term 1.
I20260926 16:31:38.502007 14156 raft_consensus.cc:2454] T ade0aa407af64d9ea312e7fcdb75bc77 P 338f775607c644ba8be145b78f82edb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eac546f77d24ea08f44d69d57b1cac5 in term 1.
I20260926 16:31:38.502432 14224 leader_election.cc:302] T ade0aa407af64d9ea312e7fcdb75bc77 P 1eac546f77d24ea08f44d69d57b1cac5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1eac546f77d24ea08f44d69d57b1cac5, 338f775607c644ba8be145b78f82edb7, ee5950d438ab4f53bab4f057a1712254; no voters:
I20260926 16:31:38.502738 14372 raft_consensus.cc:2781] T ade0aa407af64d9ea312e7fcdb75bc77 P 1eac546f77d24ea08f44d69d57b1cac5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:31:38.503244 14372 raft_consensus.cc:686] T ade0aa407af64d9ea312e7fcdb75bc77 P 1eac546f77d24ea08f44d69d57b1cac5 [term 1 LEADER]: Becoming Leader. State: Replica: 1eac546f77d24ea08f44d69d57b1cac5, State: Running, Role: LEADER
I20260926 16:31:38.503773 14372 consensus_queue.cc:237] T ade0aa407af64d9ea312e7fcdb75bc77 P 1eac546f77d24ea08f44d69d57b1cac5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eac546f77d24ea08f44d69d57b1cac5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39807 } } peers { permanent_uuid: "ee5950d438ab4f53bab4f057a1712254" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 32903 } } peers { permanent_uuid: "338f775607c644ba8be145b78f82edb7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36057 } }
I20260926 16:31:38.508216 13855 catalog_manager.cc:5495] T ade0aa407af64d9ea312e7fcdb75bc77 P 1eac546f77d24ea08f44d69d57b1cac5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1eac546f77d24ea08f44d69d57b1cac5 (127.6.33.3). New cstate: current_term: 1 leader_uuid: "1eac546f77d24ea08f44d69d57b1cac5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eac546f77d24ea08f44d69d57b1cac5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee5950d438ab4f53bab4f057a1712254" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 32903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "338f775607c644ba8be145b78f82edb7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36057 } health_report { overall_health: UNKNOWN } } }
W20260926 16:31:38.517196 14202 tablet.cc:2369] T ade0aa407af64d9ea312e7fcdb75bc77 P 338f775607c644ba8be145b78f82edb7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:31:38.528107 14361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:38.522873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60294 (local address 127.6.33.3:39807)
0926 16:31:38.523052 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:38.523056 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:38.523167 (+ 111us) server_negotiation.cc:407] Connection header received
0926 16:31:38.523360 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:38.523364 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:38.523439 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:38.523559 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:31:38.523854 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:38.524638 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:38.526196 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:38.526605 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:38.526697 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:38.527082 (+ 385us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:31:38.527664 (+ 582us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:31:38.527770 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:38.527838 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 16:31:38.527906 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 16:31:38.536525 14156 raft_consensus.cc:1261] T ade0aa407af64d9ea312e7fcdb75bc77 P 338f775607c644ba8be145b78f82edb7 [term 1 FOLLOWER]: Refusing update from remote peer 1eac546f77d24ea08f44d69d57b1cac5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:31:38.536525 14012 raft_consensus.cc:1261] T ade0aa407af64d9ea312e7fcdb75bc77 P ee5950d438ab4f53bab4f057a1712254 [term 1 FOLLOWER]: Refusing update from remote peer 1eac546f77d24ea08f44d69d57b1cac5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:31:38.537571 14375 consensus_queue.cc:1035] T ade0aa407af64d9ea312e7fcdb75bc77 P 1eac546f77d24ea08f44d69d57b1cac5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee5950d438ab4f53bab4f057a1712254" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 32903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:31:38.537976 14372 consensus_queue.cc:1035] T ade0aa407af64d9ea312e7fcdb75bc77 P 1eac546f77d24ea08f44d69d57b1cac5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "338f775607c644ba8be145b78f82edb7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:31:38.549185 14378 mvcc.cc:205] Tried to move back new op lower bound from 7333643462795218944 to 7333643462675648512. Current Snapshot: MvccSnapshot[applied={T|T < 7333643462795218944}]
I20260926 16:31:38.565174 13854 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:31:38.565553 13854 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:31:38.569425 13854 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f9211160536747d4bb0d0bceaaae9ffb: Sending DeleteTablet for 3 replicas of tablet ade0aa407af64d9ea312e7fcdb75bc77
I20260926 16:31:38.570598 14271 tablet_service.cc:1514] Processing DeleteTablet for tablet ade0aa407af64d9ea312e7fcdb75bc77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:31:38 UTC) from {username='oryx', principal='oryx/127.6.33.62@KRBTEST.COM'} at 127.0.0.1:60286
I20260926 16:31:38.570791 13992 tablet_service.cc:1514] Processing DeleteTablet for tablet ade0aa407af64d9ea312e7fcdb75bc77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:31:38 UTC) from {username='oryx', principal='oryx/127.6.33.62@KRBTEST.COM'} at 127.0.0.1:44908
I20260926 16:31:38.571033 14131 tablet_service.cc:1514] Processing DeleteTablet for tablet ade0aa407af64d9ea312e7fcdb75bc77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:31:38 UTC) from {username='oryx', principal='oryx/127.6.33.62@KRBTEST.COM'} at 127.0.0.1:58294
I20260926 16:31:38.574342 13854 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44824:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:31:38.575165 13854 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:31:38.585984 14390 tablet_replica.cc:330] T ade0aa407af64d9ea312e7fcdb75bc77 P 338f775607c644ba8be145b78f82edb7: stopping tablet replica
I20260926 16:31:38.586073 14389 tablet_replica.cc:330] T ade0aa407af64d9ea312e7fcdb75bc77 P ee5950d438ab4f53bab4f057a1712254: stopping tablet replica
I20260926 16:31:38.586573 14389 raft_consensus.cc:2229] T ade0aa407af64d9ea312e7fcdb75bc77 P ee5950d438ab4f53bab4f057a1712254 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:31:38.587112 14389 raft_consensus.cc:2258] T ade0aa407af64d9ea312e7fcdb75bc77 P ee5950d438ab4f53bab4f057a1712254 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:31:38.588624 13992 tablet_service.cc:1467] Processing CreateTablet for tablet 1175a16c0e204c979870968493a33a66 (DEFAULT_TABLE table=test-table [id=8baa762cd4a14e0fa14b9fe4cd16d466]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:31:38.588652 14270 tablet_service.cc:1467] Processing CreateTablet for tablet 1175a16c0e204c979870968493a33a66 (DEFAULT_TABLE table=test-table [id=8baa762cd4a14e0fa14b9fe4cd16d466]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:31:38.588619 14131 tablet_service.cc:1467] Processing CreateTablet for tablet 1175a16c0e204c979870968493a33a66 (DEFAULT_TABLE table=test-table [id=8baa762cd4a14e0fa14b9fe4cd16d466]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:31:38.589107 13992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1175a16c0e204c979870968493a33a66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:31:38.589169 14270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1175a16c0e204c979870968493a33a66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:31:38.589320 14131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1175a16c0e204c979870968493a33a66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:31:38.589496 14389 ts_tablet_manager.cc:1905] T ade0aa407af64d9ea312e7fcdb75bc77 P ee5950d438ab4f53bab4f057a1712254: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:31:38.595901 14366 tablet_bootstrap.cc:492] T 1175a16c0e204c979870968493a33a66 P ee5950d438ab4f53bab4f057a1712254: Bootstrap starting.
I20260926 16:31:38.595929 14365 tablet_bootstrap.cc:492] T 1175a16c0e204c979870968493a33a66 P 1eac546f77d24ea08f44d69d57b1cac5: Bootstrap starting.
I20260926 16:31:38.597007 14390 raft_consensus.cc:2229] T ade0aa407af64d9ea312e7fcdb75bc77 P 338f775607c644ba8be145b78f82edb7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:31:38.597599 14366 tablet_bootstrap.cc:654] T 1175a16c0e204c979870968493a33a66 P ee5950d438ab4f53bab4f057a1712254: Neither blocks nor log segments found. Creating new log.
I20260926 16:31:38.597616 14365 tablet_bootstrap.cc:654] T 1175a16c0e204c979870968493a33a66 P 1eac546f77d24ea08f44d69d57b1cac5: Neither blocks nor log segments found. Creating new log.
I20260926 16:31:38.597630 14390 raft_consensus.cc:2258] T ade0aa407af64d9ea312e7fcdb75bc77 P 338f775607c644ba8be145b78f82edb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:31:38.602092 14390 ts_tablet_manager.cc:1905] T ade0aa407af64d9ea312e7fcdb75bc77 P 338f775607c644ba8be145b78f82edb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:31:38.605129 14367 tablet_bootstrap.cc:492] T 1175a16c0e204c979870968493a33a66 P 338f775607c644ba8be145b78f82edb7: Bootstrap starting.
I20260926 16:31:38.607194 14367 tablet_bootstrap.cc:654] T 1175a16c0e204c979870968493a33a66 P 338f775607c644ba8be145b78f82edb7: Neither blocks nor log segments found. Creating new log.
I20260926 16:31:38.609138 14365 tablet_bootstrap.cc:492] T 1175a16c0e204c979870968493a33a66 P 1eac546f77d24ea08f44d69d57b1cac5: No bootstrap required, opened a new log
I20260926 16:31:38.609326 14365 ts_tablet_manager.cc:1397] T 1175a16c0e204c979870968493a33a66 P 1eac546f77d24ea08f44d69d57b1cac5: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20260926 16:31:38.610391 14365 raft_consensus.cc:348] T 1175a16c0e204c979870968493a33a66 P 1eac546f77d24ea08f44d69d57b1cac5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5950d438ab4f53bab4f057a1712254" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 32903 } } peers { permanent_uuid: "338f775607c644ba8be145b78f82edb7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36057 } } peers { permanent_uuid: "1eac546f77d24ea08f44d69d57b1cac5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39807 } }
I20260926 16:31:38.610684 14365 raft_consensus.cc:374] T 1175a16c0e204c979870968493a33a66 P 1eac546f77d24ea08f44d69d57b1cac5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:31:38.610805 14365 raft_consensus.cc:729] T 1175a16c0e204c979870968493a33a66 P 1eac546f77d24ea08f44d69d57b1cac5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eac546f77d24ea08f44d69d57b1cac5, State: Initialized, Role: FOLLOWER
I20260926 16:31:38.611114 14365 consensus_queue.cc:260] T 1175a16c0e204c979870968493a33a66 P 1eac546f77d24ea08f44d69d57b1cac5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5950d438ab4f53bab4f057a1712254" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 32903 } } peers { permanent_uuid: "338f775607c644ba8be145b78f82edb7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36057 } } peers { permanent_uuid: "1eac546f77d24ea08f44d69d57b1cac5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39807 } }
I20260926 16:31:38.612175 14365 ts_tablet_manager.cc:1428] T 1175a16c0e204c979870968493a33a66 P 1eac546f77d24ea08f44d69d57b1cac5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 16:31:38.614533 14390 ts_tablet_manager.cc:1918] T ade0aa407af64d9ea312e7fcdb75bc77 P 338f775607c644ba8be145b78f82edb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:31:38.614701 14390 log.cc:1196] T ade0aa407af64d9ea312e7fcdb75bc77 P 338f775607c644ba8be145b78f82edb7: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/ts-1/wal/wals/ade0aa407af64d9ea312e7fcdb75bc77
I20260926 16:31:38.615239 14390 ts_tablet_manager.cc:1939] T ade0aa407af64d9ea312e7fcdb75bc77 P 338f775607c644ba8be145b78f82edb7: Deleting consensus metadata
I20260926 16:31:38.617486 14366 tablet_bootstrap.cc:492] T 1175a16c0e204c979870968493a33a66 P ee5950d438ab4f53bab4f057a1712254: No bootstrap required, opened a new log
I20260926 16:31:38.617669 14366 ts_tablet_manager.cc:1397] T 1175a16c0e204c979870968493a33a66 P ee5950d438ab4f53bab4f057a1712254: Time spent bootstrapping tablet: real 0.022s user 0.008s sys 0.000s
I20260926 16:31:38.618346 14366 raft_consensus.cc:348] T 1175a16c0e204c979870968493a33a66 P ee5950d438ab4f53bab4f057a1712254 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5950d438ab4f53bab4f057a1712254" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 32903 } } peers { permanent_uuid: "338f775607c644ba8be145b78f82edb7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36057 } } peers { permanent_uuid: "1eac546f77d24ea08f44d69d57b1cac5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39807 } }
I20260926 16:31:38.618549 14366 raft_consensus.cc:374] T 1175a16c0e204c979870968493a33a66 P ee5950d438ab4f53bab4f057a1712254 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:31:38.618644 14366 raft_consensus.cc:729] T 1175a16c0e204c979870968493a33a66 P ee5950d438ab4f53bab4f057a1712254 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee5950d438ab4f53bab4f057a1712254, State: Initialized, Role: FOLLOWER
I20260926 16:31:38.618934 14366 consensus_queue.cc:260] T 1175a16c0e204c979870968493a33a66 P ee5950d438ab4f53bab4f057a1712254 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5950d438ab4f53bab4f057a1712254" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 32903 } } peers { permanent_uuid: "338f775607c644ba8be145b78f82edb7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36057 } } peers { permanent_uuid: "1eac546f77d24ea08f44d69d57b1cac5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39807 } }
I20260926 16:31:38.619482 14366 ts_tablet_manager.cc:1428] T 1175a16c0e204c979870968493a33a66 P ee5950d438ab4f53bab4f057a1712254: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 16:31:38.619902 13842 catalog_manager.cc:4841] TS 338f775607c644ba8be145b78f82edb7 (127.6.33.2:36057): tablet ade0aa407af64d9ea312e7fcdb75bc77 (table test-table [id=585ef87554a446edaba01dfe192c1dfe]) successfully deleted
I20260926 16:31:38.626417 14375 raft_consensus.cc:482] T 1175a16c0e204c979870968493a33a66 P 1eac546f77d24ea08f44d69d57b1cac5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:31:38.626469 14389 ts_tablet_manager.cc:1918] T ade0aa407af64d9ea312e7fcdb75bc77 P ee5950d438ab4f53bab4f057a1712254: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:31:38.626636 14389 log.cc:1196] T ade0aa407af64d9ea312e7fcdb75bc77 P ee5950d438ab4f53bab4f057a1712254: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/ts-0/wal/wals/ade0aa407af64d9ea312e7fcdb75bc77
I20260926 16:31:38.626624 14375 raft_consensus.cc:504] T 1175a16c0e204c979870968493a33a66 P 1eac546f77d24ea08f44d69d57b1cac5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5950d438ab4f53bab4f057a1712254" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 32903 } } peers { permanent_uuid: "338f775607c644ba8be145b78f82edb7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36057 } } peers { permanent_uuid: "1eac546f77d24ea08f44d69d57b1cac5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39807 } }
I20260926 16:31:38.627327 14375 leader_election.cc:288] T 1175a16c0e204c979870968493a33a66 P 1eac546f77d24ea08f44d69d57b1cac5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee5950d438ab4f53bab4f057a1712254 (127.6.33.1:32903), 338f775607c644ba8be145b78f82edb7 (127.6.33.2:36057)
I20260926 16:31:38.627823 14012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1175a16c0e204c979870968493a33a66" candidate_uuid: "1eac546f77d24ea08f44d69d57b1cac5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee5950d438ab4f53bab4f057a1712254" is_pre_election: true
I20260926 16:31:38.627985 14156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1175a16c0e204c979870968493a33a66" candidate_uuid: "1eac546f77d24ea08f44d69d57b1cac5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "338f775607c644ba8be145b78f82edb7" is_pre_election: true
I20260926 16:31:38.628051 14012 raft_consensus.cc:2454] T 1175a16c0e204c979870968493a33a66 P ee5950d438ab4f53bab4f057a1712254 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eac546f77d24ea08f44d69d57b1cac5 in term 0.
I20260926 16:31:38.628479 14223 leader_election.cc:302] T 1175a16c0e204c979870968493a33a66 P 1eac546f77d24ea08f44d69d57b1cac5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eac546f77d24ea08f44d69d57b1cac5, ee5950d438ab4f53bab4f057a1712254; no voters:
I20260926 16:31:38.628832 14375 raft_consensus.cc:2781] T 1175a16c0e204c979870968493a33a66 P 1eac546f77d24ea08f44d69d57b1cac5 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260926 16:31:38.628787 14224 leader_election.cc:341] T 1175a16c0e204c979870968493a33a66 P 1eac546f77d24ea08f44d69d57b1cac5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 338f775607c644ba8be145b78f82edb7 (127.6.33.2:36057): Illegal state: must be running to vote when last-logged opid is not known
I20260926 16:31:38.628928 14375 raft_consensus.cc:482] T 1175a16c0e204c979870968493a33a66 P 1eac546f77d24ea08f44d69d57b1cac5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:31:38.629048 14375 raft_consensus.cc:3037] T 1175a16c0e204c979870968493a33a66 P 1eac546f77d24ea08f44d69d57b1cac5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:31:38.630674 14367 tablet_bootstrap.cc:492] T 1175a16c0e204c979870968493a33a66 P 338f775607c644ba8be145b78f82edb7: No bootstrap required, opened a new log
I20260926 16:31:38.630913 14367 ts_tablet_manager.cc:1397] T 1175a16c0e204c979870968493a33a66 P 338f775607c644ba8be145b78f82edb7: Time spent bootstrapping tablet: real 0.026s user 0.011s sys 0.005s
I20260926 16:31:38.631721 14367 raft_consensus.cc:348] T 1175a16c0e204c979870968493a33a66 P 338f775607c644ba8be145b78f82edb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5950d438ab4f53bab4f057a1712254" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 32903 } } peers { permanent_uuid: "338f775607c644ba8be145b78f82edb7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36057 } } peers { permanent_uuid: "1eac546f77d24ea08f44d69d57b1cac5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39807 } }
I20260926 16:31:38.632030 14367 raft_consensus.cc:374] T 1175a16c0e204c979870968493a33a66 P 338f775607c644ba8be145b78f82edb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:31:38.632156 14367 raft_consensus.cc:729] T 1175a16c0e204c979870968493a33a66 P 338f775607c644ba8be145b78f82edb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 338f775607c644ba8be145b78f82edb7, State: Initialized, Role: FOLLOWER
I20260926 16:31:38.632242 14375 raft_consensus.cc:504] T 1175a16c0e204c979870968493a33a66 P 1eac546f77d24ea08f44d69d57b1cac5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5950d438ab4f53bab4f057a1712254" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 32903 } } peers { permanent_uuid: "338f775607c644ba8be145b78f82edb7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36057 } } peers { permanent_uuid: "1eac546f77d24ea08f44d69d57b1cac5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39807 } }
I20260926 16:31:38.632449 14367 consensus_queue.cc:260] T 1175a16c0e204c979870968493a33a66 P 338f775607c644ba8be145b78f82edb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5950d438ab4f53bab4f057a1712254" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 32903 } } peers { permanent_uuid: "338f775607c644ba8be145b78f82edb7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36057 } } peers { permanent_uuid: "1eac546f77d24ea08f44d69d57b1cac5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39807 } }
I20260926 16:31:38.633404 14012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1175a16c0e204c979870968493a33a66" candidate_uuid: "1eac546f77d24ea08f44d69d57b1cac5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee5950d438ab4f53bab4f057a1712254"
I20260926 16:31:38.633653 14012 raft_consensus.cc:3037] T 1175a16c0e204c979870968493a33a66 P ee5950d438ab4f53bab4f057a1712254 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:31:38.635519 14367 ts_tablet_manager.cc:1428] T 1175a16c0e204c979870968493a33a66 P 338f775607c644ba8be145b78f82edb7: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260926 16:31:38.636526 14012 raft_consensus.cc:2454] T 1175a16c0e204c979870968493a33a66 P ee5950d438ab4f53bab4f057a1712254 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eac546f77d24ea08f44d69d57b1cac5 in term 1.
I20260926 16:31:38.637044 14223 leader_election.cc:302] T 1175a16c0e204c979870968493a33a66 P 1eac546f77d24ea08f44d69d57b1cac5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eac546f77d24ea08f44d69d57b1cac5, ee5950d438ab4f53bab4f057a1712254; no voters:
I20260926 16:31:38.637369 14372 raft_consensus.cc:2781] T 1175a16c0e204c979870968493a33a66 P 1eac546f77d24ea08f44d69d57b1cac5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:31:38.637528 14372 raft_consensus.cc:686] T 1175a16c0e204c979870968493a33a66 P 1eac546f77d24ea08f44d69d57b1cac5 [term 1 LEADER]: Becoming Leader. State: Replica: 1eac546f77d24ea08f44d69d57b1cac5, State: Running, Role: LEADER
I20260926 16:31:38.637804 14372 consensus_queue.cc:237] T 1175a16c0e204c979870968493a33a66 P 1eac546f77d24ea08f44d69d57b1cac5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5950d438ab4f53bab4f057a1712254" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 32903 } } peers { permanent_uuid: "338f775607c644ba8be145b78f82edb7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36057 } } peers { permanent_uuid: "1eac546f77d24ea08f44d69d57b1cac5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39807 } }
I20260926 16:31:38.639793 14375 leader_election.cc:288] T 1175a16c0e204c979870968493a33a66 P 1eac546f77d24ea08f44d69d57b1cac5 [CANDIDATE]: Term 1 election: Requested vote from peers ee5950d438ab4f53bab4f057a1712254 (127.6.33.1:32903), 338f775607c644ba8be145b78f82edb7 (127.6.33.2:36057)
I20260926 16:31:38.640388 14156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1175a16c0e204c979870968493a33a66" candidate_uuid: "1eac546f77d24ea08f44d69d57b1cac5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "338f775607c644ba8be145b78f82edb7"
I20260926 16:31:38.640666 14156 raft_consensus.cc:3037] T 1175a16c0e204c979870968493a33a66 P 338f775607c644ba8be145b78f82edb7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:31:38.644205 14388 tablet_replica.cc:330] T ade0aa407af64d9ea312e7fcdb75bc77 P 1eac546f77d24ea08f44d69d57b1cac5: stopping tablet replica
I20260926 16:31:38.644284 14156 raft_consensus.cc:2454] T 1175a16c0e204c979870968493a33a66 P 338f775607c644ba8be145b78f82edb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eac546f77d24ea08f44d69d57b1cac5 in term 1.
I20260926 16:31:38.644673 14388 raft_consensus.cc:2229] T ade0aa407af64d9ea312e7fcdb75bc77 P 1eac546f77d24ea08f44d69d57b1cac5 [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:31:38.645514 14388 raft_consensus.cc:2258] T ade0aa407af64d9ea312e7fcdb75bc77 P 1eac546f77d24ea08f44d69d57b1cac5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:31:38.627192 14389 ts_tablet_manager.cc:1939] T ade0aa407af64d9ea312e7fcdb75bc77 P ee5950d438ab4f53bab4f057a1712254: Deleting consensus metadata
I20260926 16:31:38.647606 13855 catalog_manager.cc:5495] T 1175a16c0e204c979870968493a33a66 P 1eac546f77d24ea08f44d69d57b1cac5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1eac546f77d24ea08f44d69d57b1cac5 (127.6.33.3). New cstate: current_term: 1 leader_uuid: "1eac546f77d24ea08f44d69d57b1cac5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5950d438ab4f53bab4f057a1712254" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 32903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "338f775607c644ba8be145b78f82edb7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eac546f77d24ea08f44d69d57b1cac5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39807 } health_report { overall_health: HEALTHY } } }
I20260926 16:31:38.648075 14388 ts_tablet_manager.cc:1905] T ade0aa407af64d9ea312e7fcdb75bc77 P 1eac546f77d24ea08f44d69d57b1cac5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:31:38.648540 13841 catalog_manager.cc:4841] TS ee5950d438ab4f53bab4f057a1712254 (127.6.33.1:32903): tablet ade0aa407af64d9ea312e7fcdb75bc77 (table test-table [id=585ef87554a446edaba01dfe192c1dfe]) successfully deleted
I20260926 16:31:38.654112 14388 ts_tablet_manager.cc:1918] T ade0aa407af64d9ea312e7fcdb75bc77 P 1eac546f77d24ea08f44d69d57b1cac5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:31:38.654345 14388 log.cc:1196] T ade0aa407af64d9ea312e7fcdb75bc77 P 1eac546f77d24ea08f44d69d57b1cac5: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/ts-2/wal/wals/ade0aa407af64d9ea312e7fcdb75bc77
I20260926 16:31:38.654829 14388 ts_tablet_manager.cc:1939] T ade0aa407af64d9ea312e7fcdb75bc77 P 1eac546f77d24ea08f44d69d57b1cac5: Deleting consensus metadata
I20260926 16:31:38.656255 13840 catalog_manager.cc:4841] TS 1eac546f77d24ea08f44d69d57b1cac5 (127.6.33.3:39807): tablet ade0aa407af64d9ea312e7fcdb75bc77 (table test-table [id=585ef87554a446edaba01dfe192c1dfe]) successfully deleted
Sep 26 16:31:38 dist-test-slave-khd3 krb5kdc[13810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440296, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.33.62@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM
I20260926 16:31:38.711591 14400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:38.691500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:38215 (local address 127.0.0.1:44830)
0926 16:31:38.692266 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:38.692282 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:38.692452 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:38.692800 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:38.692805 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:38.693521 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:38.693820 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:38.693831 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:38.694839 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:38.694846 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:38.695836 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:38.695847 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:38.696554 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:38.696561 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:38.696726 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:38.702448 (+ 5722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:38.702471 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:38.705351 (+ 2880us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:31:38.708824 (+ 3473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:38.708840 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:38.708849 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:38.709282 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:38.709664 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:38.709669 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:38.709672 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:38.709813 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:38.710432 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:31:38.710440 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:31:38.710613 (+ 173us) client_negotiation.cc:769] Sending connection context
0926 16:31:38.710792 (+ 179us) client_negotiation.cc:240] Negotiation successful
0926 16:31:38.711127 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":464,"threads_started":1}
I20260926 16:31:38.711941 14230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:38.691486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44830 (local address 127.6.33.62:38215)
0926 16:31:38.691752 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:38.691762 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:38.692373 (+ 611us) server_negotiation.cc:407] Connection header received
0926 16:31:38.692573 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:38.692583 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:38.692636 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:38.692737 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:38.693972 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:38.694702 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:38.695979 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:38.696378 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:38.696517 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:38.705687 (+ 9170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:38.705694 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:38.705699 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:38.705762 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:38.708600 (+ 2838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:38.709410 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:38.709418 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:38.709424 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:38.709549 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:38.709921 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:38.709926 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:38.709929 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:38.710197 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:38.710440 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:38.710883 (+ 443us) server_negotiation.cc:299] Negotiation successful
0926 16:31:38.711243 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":151,"server-negotiator.queue_time_us":116}
I20260926 16:31:38.717414 13855 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.6.33.62@KRBTEST.COM'} at 127.0.0.1:44830:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 16:31:38.718441 13855 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:31:38.725268 14131 tablet_service.cc:1467] Processing CreateTablet for tablet d4b8bcc49ae74f008bb82fa68c5cc96c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=dcea62cea5574e13be6fe4d6e24f5e2e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 16:31:38.725276 13992 tablet_service.cc:1467] Processing CreateTablet for tablet d4b8bcc49ae74f008bb82fa68c5cc96c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=dcea62cea5574e13be6fe4d6e24f5e2e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 16:31:38.725579 14271 tablet_service.cc:1467] Processing CreateTablet for tablet d4b8bcc49ae74f008bb82fa68c5cc96c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=dcea62cea5574e13be6fe4d6e24f5e2e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 16:31:38.725821 14131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4b8bcc49ae74f008bb82fa68c5cc96c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:31:38.725817 13992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4b8bcc49ae74f008bb82fa68c5cc96c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:31:38.726123 14271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4b8bcc49ae74f008bb82fa68c5cc96c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:31:38.733086 14367 tablet_bootstrap.cc:492] T d4b8bcc49ae74f008bb82fa68c5cc96c P 338f775607c644ba8be145b78f82edb7: Bootstrap starting.
I20260926 16:31:38.733675 14366 tablet_bootstrap.cc:492] T d4b8bcc49ae74f008bb82fa68c5cc96c P ee5950d438ab4f53bab4f057a1712254: Bootstrap starting.
I20260926 16:31:38.734503 14365 tablet_bootstrap.cc:492] T d4b8bcc49ae74f008bb82fa68c5cc96c P 1eac546f77d24ea08f44d69d57b1cac5: Bootstrap starting.
I20260926 16:31:38.734894 14367 tablet_bootstrap.cc:654] T d4b8bcc49ae74f008bb82fa68c5cc96c P 338f775607c644ba8be145b78f82edb7: Neither blocks nor log segments found. Creating new log.
I20260926 16:31:38.735409 14366 tablet_bootstrap.cc:654] T d4b8bcc49ae74f008bb82fa68c5cc96c P ee5950d438ab4f53bab4f057a1712254: Neither blocks nor log segments found. Creating new log.
I20260926 16:31:38.736130 14365 tablet_bootstrap.cc:654] T d4b8bcc49ae74f008bb82fa68c5cc96c P 1eac546f77d24ea08f44d69d57b1cac5: Neither blocks nor log segments found. Creating new log.
I20260926 16:31:38.746235 14365 tablet_bootstrap.cc:492] T d4b8bcc49ae74f008bb82fa68c5cc96c P 1eac546f77d24ea08f44d69d57b1cac5: No bootstrap required, opened a new log
I20260926 16:31:38.746510 14365 ts_tablet_manager.cc:1397] T d4b8bcc49ae74f008bb82fa68c5cc96c P 1eac546f77d24ea08f44d69d57b1cac5: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260926 16:31:38.747363 14365 raft_consensus.cc:348] T d4b8bcc49ae74f008bb82fa68c5cc96c P 1eac546f77d24ea08f44d69d57b1cac5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5950d438ab4f53bab4f057a1712254" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 32903 } } peers { permanent_uuid: "338f775607c644ba8be145b78f82edb7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36057 } } peers { permanent_uuid: "1eac546f77d24ea08f44d69d57b1cac5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39807 } }
I20260926 16:31:38.747784 14366 tablet_bootstrap.cc:492] T d4b8bcc49ae74f008bb82fa68c5cc96c P ee5950d438ab4f53bab4f057a1712254: No bootstrap required, opened a new log
I20260926 16:31:38.747999 14366 ts_tablet_manager.cc:1397] T d4b8bcc49ae74f008bb82fa68c5cc96c P ee5950d438ab4f53bab4f057a1712254: Time spent bootstrapping tablet: real 0.014s user 0.012s sys 0.000s
I20260926 16:31:38.748054 14367 tablet_bootstrap.cc:492] T d4b8bcc49ae74f008bb82fa68c5cc96c P 338f775607c644ba8be145b78f82edb7: No bootstrap required, opened a new log
I20260926 16:31:38.748199 14367 ts_tablet_manager.cc:1397] T d4b8bcc49ae74f008bb82fa68c5cc96c P 338f775607c644ba8be145b78f82edb7: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.005s
I20260926 16:31:38.748811 14365 raft_consensus.cc:374] T d4b8bcc49ae74f008bb82fa68c5cc96c P 1eac546f77d24ea08f44d69d57b1cac5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:31:38.748724 14366 raft_consensus.cc:348] T d4b8bcc49ae74f008bb82fa68c5cc96c P ee5950d438ab4f53bab4f057a1712254 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5950d438ab4f53bab4f057a1712254" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 32903 } } peers { permanent_uuid: "338f775607c644ba8be145b78f82edb7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36057 } } peers { permanent_uuid: "1eac546f77d24ea08f44d69d57b1cac5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39807 } }
I20260926 16:31:38.748996 14366 raft_consensus.cc:374] T d4b8bcc49ae74f008bb82fa68c5cc96c P ee5950d438ab4f53bab4f057a1712254 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:31:38.748890 14367 raft_consensus.cc:348] T d4b8bcc49ae74f008bb82fa68c5cc96c P 338f775607c644ba8be145b78f82edb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5950d438ab4f53bab4f057a1712254" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 32903 } } peers { permanent_uuid: "338f775607c644ba8be145b78f82edb7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36057 } } peers { permanent_uuid: "1eac546f77d24ea08f44d69d57b1cac5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39807 } }
I20260926 16:31:38.749071 14366 raft_consensus.cc:729] T d4b8bcc49ae74f008bb82fa68c5cc96c P ee5950d438ab4f53bab4f057a1712254 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee5950d438ab4f53bab4f057a1712254, State: Initialized, Role: FOLLOWER
I20260926 16:31:38.749011 14365 raft_consensus.cc:729] T d4b8bcc49ae74f008bb82fa68c5cc96c P 1eac546f77d24ea08f44d69d57b1cac5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eac546f77d24ea08f44d69d57b1cac5, State: Initialized, Role: FOLLOWER
I20260926 16:31:38.749116 14367 raft_consensus.cc:374] T d4b8bcc49ae74f008bb82fa68c5cc96c P 338f775607c644ba8be145b78f82edb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:31:38.749191 14367 raft_consensus.cc:729] T d4b8bcc49ae74f008bb82fa68c5cc96c P 338f775607c644ba8be145b78f82edb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 338f775607c644ba8be145b78f82edb7, State: Initialized, Role: FOLLOWER
I20260926 16:31:38.749392 14366 consensus_queue.cc:260] T d4b8bcc49ae74f008bb82fa68c5cc96c P ee5950d438ab4f53bab4f057a1712254 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5950d438ab4f53bab4f057a1712254" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 32903 } } peers { permanent_uuid: "338f775607c644ba8be145b78f82edb7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36057 } } peers { permanent_uuid: "1eac546f77d24ea08f44d69d57b1cac5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39807 } }
I20260926 16:31:38.749487 14367 consensus_queue.cc:260] T d4b8bcc49ae74f008bb82fa68c5cc96c P 338f775607c644ba8be145b78f82edb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5950d438ab4f53bab4f057a1712254" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 32903 } } peers { permanent_uuid: "338f775607c644ba8be145b78f82edb7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36057 } } peers { permanent_uuid: "1eac546f77d24ea08f44d69d57b1cac5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39807 } }
I20260926 16:31:38.749995 14367 ts_tablet_manager.cc:1428] T d4b8bcc49ae74f008bb82fa68c5cc96c P 338f775607c644ba8be145b78f82edb7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 16:31:38.749990 14366 ts_tablet_manager.cc:1428] T d4b8bcc49ae74f008bb82fa68c5cc96c P ee5950d438ab4f53bab4f057a1712254: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260926 16:31:38.749897 14371 tablet_replica.cc:439] T d4b8bcc49ae74f008bb82fa68c5cc96c P ee5950d438ab4f53bab4f057a1712254: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5950d438ab4f53bab4f057a1712254" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 32903 } } peers { permanent_uuid: "338f775607c644ba8be145b78f82edb7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36057 } } peers { permanent_uuid: "1eac546f77d24ea08f44d69d57b1cac5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39807 } } }
I20260926 16:31:38.750249 14371 tablet_replica.cc:442] T d4b8bcc49ae74f008bb82fa68c5cc96c P ee5950d438ab4f53bab4f057a1712254: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:31:38.750571 14373 tablet_replica.cc:439] T d4b8bcc49ae74f008bb82fa68c5cc96c P 338f775607c644ba8be145b78f82edb7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5950d438ab4f53bab4f057a1712254" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 32903 } } peers { permanent_uuid: "338f775607c644ba8be145b78f82edb7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36057 } } peers { permanent_uuid: "1eac546f77d24ea08f44d69d57b1cac5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39807 } } }
I20260926 16:31:38.750901 14373 tablet_replica.cc:442] T d4b8bcc49ae74f008bb82fa68c5cc96c P 338f775607c644ba8be145b78f82edb7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:31:38.749454 14365 consensus_queue.cc:260] T d4b8bcc49ae74f008bb82fa68c5cc96c P 1eac546f77d24ea08f44d69d57b1cac5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5950d438ab4f53bab4f057a1712254" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 32903 } } peers { permanent_uuid: "338f775607c644ba8be145b78f82edb7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36057 } } peers { permanent_uuid: "1eac546f77d24ea08f44d69d57b1cac5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39807 } }
I20260926 16:31:38.751994 14372 tablet_replica.cc:439] T d4b8bcc49ae74f008bb82fa68c5cc96c P 1eac546f77d24ea08f44d69d57b1cac5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5950d438ab4f53bab4f057a1712254" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 32903 } } peers { permanent_uuid: "338f775607c644ba8be145b78f82edb7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36057 } } peers { permanent_uuid: "1eac546f77d24ea08f44d69d57b1cac5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39807 } } }
I20260926 16:31:38.752310 14372 tablet_replica.cc:442] T d4b8bcc49ae74f008bb82fa68c5cc96c P 1eac546f77d24ea08f44d69d57b1cac5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:31:38.753997 14365 ts_tablet_manager.cc:1428] T d4b8bcc49ae74f008bb82fa68c5cc96c P 1eac546f77d24ea08f44d69d57b1cac5: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
W20260926 16:31:38.804589 14342 tablet.cc:2369] T 1175a16c0e204c979870968493a33a66 P 1eac546f77d24ea08f44d69d57b1cac5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:31:38.999303 14371 raft_consensus.cc:482] T d4b8bcc49ae74f008bb82fa68c5cc96c P ee5950d438ab4f53bab4f057a1712254 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:31:38.999568 14371 raft_consensus.cc:504] T d4b8bcc49ae74f008bb82fa68c5cc96c P ee5950d438ab4f53bab4f057a1712254 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5950d438ab4f53bab4f057a1712254" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 32903 } } peers { permanent_uuid: "338f775607c644ba8be145b78f82edb7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36057 } } peers { permanent_uuid: "1eac546f77d24ea08f44d69d57b1cac5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39807 } }
I20260926 16:31:39.001494 14371 leader_election.cc:288] T d4b8bcc49ae74f008bb82fa68c5cc96c P ee5950d438ab4f53bab4f057a1712254 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 338f775607c644ba8be145b78f82edb7 (127.6.33.2:36057), 1eac546f77d24ea08f44d69d57b1cac5 (127.6.33.3:39807)
I20260926 16:31:39.009527 14405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:39.002336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:51999 (local address 127.6.33.2:36057)
0926 16:31:39.002946 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:39.002952 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:39.002970 (+ 18us) server_negotiation.cc:407] Connection header received
0926 16:31:39.003039 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:39.003045 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:39.003115 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:39.003212 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:31:39.003891 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:39.005018 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:39.007405 (+ 2387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:39.008459 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:39.008588 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:39.008676 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:39.009066 (+ 390us) server_negotiation.cc:299] Negotiation successful
0926 16:31:39.009198 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":393,"threads_started":1}
I20260926 16:31:39.009559 14404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:39.001836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.2:36057 (local address 127.6.33.1:51999)
0926 16:31:39.002365 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:39.002386 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:39.002544 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:39.003244 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:39.003253 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:39.003286 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:31:39.003626 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:39.003692 (+ 66us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:39.005152 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:39.005158 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:39.007204 (+ 2046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:39.007219 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:39.008639 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:39.008645 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:39.008871 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:39.008911 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 16:31:39.009006 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 16:31:39.009154 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"mutex_wait_us":1,"thread_start_us":279,"threads_started":1}
I20260926 16:31:39.010102 14361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:39.001898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:40559 (local address 127.6.33.3:39807)
0926 16:31:39.002182 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:39.002189 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:39.003369 (+ 1180us) server_negotiation.cc:407] Connection header received
0926 16:31:39.003651 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:39.003658 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:39.003740 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:39.003836 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:31:39.004584 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:39.005642 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:39.007822 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:39.008710 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:39.008889 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:39.008980 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:39.009365 (+ 385us) server_negotiation.cc:299] Negotiation successful
0926 16:31:39.009499 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260926 16:31:39.010215 14156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4b8bcc49ae74f008bb82fa68c5cc96c" candidate_uuid: "ee5950d438ab4f53bab4f057a1712254" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "338f775607c644ba8be145b78f82edb7" is_pre_election: true
I20260926 16:31:39.010440 14406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:39.001836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.3:39807 (local address 127.6.33.1:40559)
0926 16:31:39.003262 (+ 1426us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:39.003278 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:39.003490 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:39.003920 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:39.003924 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:39.003965 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:31:39.004350 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:39.004363 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:39.005800 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:39.005806 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:39.007657 (+ 1851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:39.007667 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:39.008973 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:39.008981 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:39.009162 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:39.009194 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 16:31:39.009921 (+ 727us) client_negotiation.cc:240] Negotiation successful
0926 16:31:39.010049 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1309,"mutex_wait_us":480,"thread_start_us":684,"threads_started":1}
I20260926 16:31:39.010514 14156 raft_consensus.cc:2454] T d4b8bcc49ae74f008bb82fa68c5cc96c P 338f775607c644ba8be145b78f82edb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee5950d438ab4f53bab4f057a1712254 in term 0.
I20260926 16:31:39.011044 13945 leader_election.cc:302] T d4b8bcc49ae74f008bb82fa68c5cc96c P ee5950d438ab4f53bab4f057a1712254 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 338f775607c644ba8be145b78f82edb7, ee5950d438ab4f53bab4f057a1712254; no voters:
I20260926 16:31:39.011380 14295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4b8bcc49ae74f008bb82fa68c5cc96c" candidate_uuid: "ee5950d438ab4f53bab4f057a1712254" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eac546f77d24ea08f44d69d57b1cac5" is_pre_election: true
I20260926 16:31:39.011523 14371 raft_consensus.cc:2781] T d4b8bcc49ae74f008bb82fa68c5cc96c P ee5950d438ab4f53bab4f057a1712254 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:31:39.011662 14371 raft_consensus.cc:482] T d4b8bcc49ae74f008bb82fa68c5cc96c P ee5950d438ab4f53bab4f057a1712254 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:31:39.011777 14371 raft_consensus.cc:3037] T d4b8bcc49ae74f008bb82fa68c5cc96c P ee5950d438ab4f53bab4f057a1712254 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:31:39.011806 14295 raft_consensus.cc:2454] T d4b8bcc49ae74f008bb82fa68c5cc96c P 1eac546f77d24ea08f44d69d57b1cac5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee5950d438ab4f53bab4f057a1712254 in term 0.
I20260926 16:31:39.015137 14371 raft_consensus.cc:504] T d4b8bcc49ae74f008bb82fa68c5cc96c P ee5950d438ab4f53bab4f057a1712254 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5950d438ab4f53bab4f057a1712254" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 32903 } } peers { permanent_uuid: "338f775607c644ba8be145b78f82edb7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36057 } } peers { permanent_uuid: "1eac546f77d24ea08f44d69d57b1cac5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39807 } }
I20260926 16:31:39.015841 14371 leader_election.cc:288] T d4b8bcc49ae74f008bb82fa68c5cc96c P ee5950d438ab4f53bab4f057a1712254 [CANDIDATE]: Term 1 election: Requested vote from peers 338f775607c644ba8be145b78f82edb7 (127.6.33.2:36057), 1eac546f77d24ea08f44d69d57b1cac5 (127.6.33.3:39807)
I20260926 16:31:39.016414 14156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4b8bcc49ae74f008bb82fa68c5cc96c" candidate_uuid: "ee5950d438ab4f53bab4f057a1712254" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "338f775607c644ba8be145b78f82edb7"
I20260926 16:31:39.016433 14295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4b8bcc49ae74f008bb82fa68c5cc96c" candidate_uuid: "ee5950d438ab4f53bab4f057a1712254" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eac546f77d24ea08f44d69d57b1cac5"
I20260926 16:31:39.016623 14156 raft_consensus.cc:3037] T d4b8bcc49ae74f008bb82fa68c5cc96c P 338f775607c644ba8be145b78f82edb7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:31:39.016654 14295 raft_consensus.cc:3037] T d4b8bcc49ae74f008bb82fa68c5cc96c P 1eac546f77d24ea08f44d69d57b1cac5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:31:39.019124 14295 raft_consensus.cc:2454] T d4b8bcc49ae74f008bb82fa68c5cc96c P 1eac546f77d24ea08f44d69d57b1cac5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee5950d438ab4f53bab4f057a1712254 in term 1.
I20260926 16:31:39.019635 13943 leader_election.cc:302] T d4b8bcc49ae74f008bb82fa68c5cc96c P ee5950d438ab4f53bab4f057a1712254 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eac546f77d24ea08f44d69d57b1cac5, ee5950d438ab4f53bab4f057a1712254; no voters:
I20260926 16:31:39.019981 14371 raft_consensus.cc:2781] T d4b8bcc49ae74f008bb82fa68c5cc96c P ee5950d438ab4f53bab4f057a1712254 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:31:39.020025 14156 raft_consensus.cc:2454] T d4b8bcc49ae74f008bb82fa68c5cc96c P 338f775607c644ba8be145b78f82edb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee5950d438ab4f53bab4f057a1712254 in term 1.
I20260926 16:31:39.020452 14371 raft_consensus.cc:686] T d4b8bcc49ae74f008bb82fa68c5cc96c P ee5950d438ab4f53bab4f057a1712254 [term 1 LEADER]: Becoming Leader. State: Replica: ee5950d438ab4f53bab4f057a1712254, State: Running, Role: LEADER
I20260926 16:31:39.020934 14371 consensus_queue.cc:237] T d4b8bcc49ae74f008bb82fa68c5cc96c P ee5950d438ab4f53bab4f057a1712254 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5950d438ab4f53bab4f057a1712254" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 32903 } } peers { permanent_uuid: "338f775607c644ba8be145b78f82edb7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36057 } } peers { permanent_uuid: "1eac546f77d24ea08f44d69d57b1cac5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39807 } }
I20260926 16:31:39.023504 14407 tablet_replica.cc:439] T d4b8bcc49ae74f008bb82fa68c5cc96c P ee5950d438ab4f53bab4f057a1712254: TxnStatusTablet state changed. Reason: New leader ee5950d438ab4f53bab4f057a1712254. Latest consensus state: current_term: 1 leader_uuid: "ee5950d438ab4f53bab4f057a1712254" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5950d438ab4f53bab4f057a1712254" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 32903 } } peers { permanent_uuid: "338f775607c644ba8be145b78f82edb7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36057 } } peers { permanent_uuid: "1eac546f77d24ea08f44d69d57b1cac5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39807 } } }
I20260926 16:31:39.023844 14407 tablet_replica.cc:442] T d4b8bcc49ae74f008bb82fa68c5cc96c P ee5950d438ab4f53bab4f057a1712254: This TxnStatusTablet replica's current role is: LEADER
I20260926 16:31:39.024559 14409 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 16:31:39.024823 14409 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 16:31:39.025027 13853 catalog_manager.cc:5495] T d4b8bcc49ae74f008bb82fa68c5cc96c P ee5950d438ab4f53bab4f057a1712254 reported cstate change: term changed from 0 to 1, leader changed from <none> to ee5950d438ab4f53bab4f057a1712254 (127.6.33.1). New cstate: current_term: 1 leader_uuid: "ee5950d438ab4f53bab4f057a1712254" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5950d438ab4f53bab4f057a1712254" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 32903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "338f775607c644ba8be145b78f82edb7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eac546f77d24ea08f44d69d57b1cac5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39807 } health_report { overall_health: UNKNOWN } } }
I20260926 16:31:39.055886 14412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:39.046229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44918 (local address 127.6.33.1:32903)
0926 16:31:39.049239 (+ 3010us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:39.049245 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:39.049271 (+ 26us) server_negotiation.cc:407] Connection header received
0926 16:31:39.049374 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:39.049379 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:39.049508 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:39.049729 (+ 221us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:31:39.050213 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:39.051495 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:39.053302 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:39.053839 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:39.053978 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:39.054534 (+ 556us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:31:39.055227 (+ 693us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:31:39.055392 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:39.055519 (+ 127us) server_negotiation.cc:299] Negotiation successful
0926 16:31:39.055587 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2866,"thread_start_us":2694,"threads_started":1}
I20260926 16:31:39.055979 14400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:39.046120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:32903 (local address 127.0.0.1:44918)
0926 16:31:39.046514 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:39.046537 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:39.046682 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:39.049798 (+ 3116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:39.049804 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:39.049817 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 16:31:39.050015 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:39.050025 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:39.051628 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:39.051633 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:39.053148 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:39.053161 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:39.054027 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:39.054033 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:39.054236 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:39.054319 (+ 83us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 16:31:39.055364 (+ 1045us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 16:31:39.055387 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 16:31:39.055478 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 16:31:39.055644 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223}
I20260926 16:31:39.060777 14156 raft_consensus.cc:1261] T d4b8bcc49ae74f008bb82fa68c5cc96c P 338f775607c644ba8be145b78f82edb7 [term 1 FOLLOWER]: Refusing update from remote peer ee5950d438ab4f53bab4f057a1712254: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:31:39.061673 14295 raft_consensus.cc:1261] T d4b8bcc49ae74f008bb82fa68c5cc96c P 1eac546f77d24ea08f44d69d57b1cac5 [term 1 FOLLOWER]: Refusing update from remote peer ee5950d438ab4f53bab4f057a1712254: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:31:39.062505 14407 consensus_queue.cc:1035] T d4b8bcc49ae74f008bb82fa68c5cc96c P ee5950d438ab4f53bab4f057a1712254 [LEADER]: Connected to new peer: Peer: permanent_uuid: "338f775607c644ba8be145b78f82edb7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:31:39.063180 14407 consensus_queue.cc:1035] T d4b8bcc49ae74f008bb82fa68c5cc96c P ee5950d438ab4f53bab4f057a1712254 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eac546f77d24ea08f44d69d57b1cac5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:31:39.066633 14373 tablet_replica.cc:439] T d4b8bcc49ae74f008bb82fa68c5cc96c P 338f775607c644ba8be145b78f82edb7: TxnStatusTablet state changed. Reason: New leader ee5950d438ab4f53bab4f057a1712254. Latest consensus state: current_term: 1 leader_uuid: "ee5950d438ab4f53bab4f057a1712254" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5950d438ab4f53bab4f057a1712254" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 32903 } } peers { permanent_uuid: "338f775607c644ba8be145b78f82edb7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36057 } } peers { permanent_uuid: "1eac546f77d24ea08f44d69d57b1cac5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39807 } } }
I20260926 16:31:39.066946 14373 tablet_replica.cc:442] T d4b8bcc49ae74f008bb82fa68c5cc96c P 338f775607c644ba8be145b78f82edb7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:31:39.068781 14375 consensus_queue.cc:1035] T 1175a16c0e204c979870968493a33a66 P 1eac546f77d24ea08f44d69d57b1cac5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee5950d438ab4f53bab4f057a1712254" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 32903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:31:39.069631 14413 mvcc.cc:205] Tried to move back new op lower bound from 7333643464945680384 to 7333643464793473024. Current Snapshot: MvccSnapshot[applied={T|T < 7333643464945680384}]
I20260926 16:31:39.081352 14372 tablet_replica.cc:439] T d4b8bcc49ae74f008bb82fa68c5cc96c P 1eac546f77d24ea08f44d69d57b1cac5: TxnStatusTablet state changed. Reason: New leader ee5950d438ab4f53bab4f057a1712254. Latest consensus state: current_term: 1 leader_uuid: "ee5950d438ab4f53bab4f057a1712254" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5950d438ab4f53bab4f057a1712254" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 32903 } } peers { permanent_uuid: "338f775607c644ba8be145b78f82edb7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36057 } } peers { permanent_uuid: "1eac546f77d24ea08f44d69d57b1cac5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39807 } } }
I20260926 16:31:39.081676 14372 tablet_replica.cc:442] T d4b8bcc49ae74f008bb82fa68c5cc96c P 1eac546f77d24ea08f44d69d57b1cac5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:31:39.086884 14372 consensus_queue.cc:1035] T 1175a16c0e204c979870968493a33a66 P 1eac546f77d24ea08f44d69d57b1cac5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "338f775607c644ba8be145b78f82edb7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:31:39.087399 14372 tablet_replica.cc:439] T d4b8bcc49ae74f008bb82fa68c5cc96c P 1eac546f77d24ea08f44d69d57b1cac5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ee5950d438ab4f53bab4f057a1712254" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5950d438ab4f53bab4f057a1712254" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 32903 } } peers { permanent_uuid: "338f775607c644ba8be145b78f82edb7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36057 } } peers { permanent_uuid: "1eac546f77d24ea08f44d69d57b1cac5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39807 } } }
I20260926 16:31:39.087613 14372 tablet_replica.cc:442] T d4b8bcc49ae74f008bb82fa68c5cc96c P 1eac546f77d24ea08f44d69d57b1cac5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:31:39.099529 14373 tablet_replica.cc:439] T d4b8bcc49ae74f008bb82fa68c5cc96c P 338f775607c644ba8be145b78f82edb7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ee5950d438ab4f53bab4f057a1712254" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5950d438ab4f53bab4f057a1712254" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 32903 } } peers { permanent_uuid: "338f775607c644ba8be145b78f82edb7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36057 } } peers { permanent_uuid: "1eac546f77d24ea08f44d69d57b1cac5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39807 } } }
I20260926 16:31:39.099891 14373 tablet_replica.cc:442] T d4b8bcc49ae74f008bb82fa68c5cc96c P 338f775607c644ba8be145b78f82edb7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:31:39.105656 14407 tablet_replica.cc:439] T d4b8bcc49ae74f008bb82fa68c5cc96c P ee5950d438ab4f53bab4f057a1712254: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ee5950d438ab4f53bab4f057a1712254" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5950d438ab4f53bab4f057a1712254" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 32903 } } peers { permanent_uuid: "338f775607c644ba8be145b78f82edb7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36057 } } peers { permanent_uuid: "1eac546f77d24ea08f44d69d57b1cac5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39807 } } }
I20260926 16:31:39.105950 14407 tablet_replica.cc:442] T d4b8bcc49ae74f008bb82fa68c5cc96c P ee5950d438ab4f53bab4f057a1712254: This TxnStatusTablet replica's current role is: LEADER
I20260926 16:31:39.106575 14407 tablet_replica.cc:439] T d4b8bcc49ae74f008bb82fa68c5cc96c P ee5950d438ab4f53bab4f057a1712254: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ee5950d438ab4f53bab4f057a1712254" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5950d438ab4f53bab4f057a1712254" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 32903 } } peers { permanent_uuid: "338f775607c644ba8be145b78f82edb7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36057 } } peers { permanent_uuid: "1eac546f77d24ea08f44d69d57b1cac5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39807 } } }
I20260926 16:31:39.106833 14407 tablet_replica.cc:442] T d4b8bcc49ae74f008bb82fa68c5cc96c P ee5950d438ab4f53bab4f057a1712254: This TxnStatusTablet replica's current role is: LEADER
I20260926 16:31:39.146597 14412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:39.140497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.3:41407 (local address 127.6.33.1:32903)
0926 16:31:39.140712 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:39.140719 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:39.141346 (+ 627us) server_negotiation.cc:407] Connection header received
0926 16:31:39.141514 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:39.141519 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:39.141607 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:39.141708 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:31:39.142121 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:39.143255 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:39.144618 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:39.145304 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:39.145430 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:39.145853 (+ 423us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:31:39.145981 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:31:39.146069 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:39.146199 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 16:31:39.146284 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 16:31:39.146616 14426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:39.140507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:32903 (local address 127.6.33.3:41407)
0926 16:31:39.141232 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:39.141258 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:39.141384 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:39.141726 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:39.141729 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:39.141738 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 16:31:39.141969 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:39.141977 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:39.143404 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:39.143408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:39.144477 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:39.144490 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:39.145488 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:39.145493 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:39.145626 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:39.145683 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 16:31:39.146079 (+ 396us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 16:31:39.146088 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 16:31:39.146152 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 16:31:39.146256 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":414,"threads_started":1}
I20260926 16:31:39.201793 14361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:39.194415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:58447 (local address 127.6.33.3:39807)
0926 16:31:39.194644 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:39.194653 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:39.195435 (+ 782us) server_negotiation.cc:407] Connection header received
0926 16:31:39.195571 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:39.195575 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:39.195661 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:39.195775 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:31:39.196621 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:39.197671 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:39.199561 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:39.200238 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:39.200381 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:39.200966 (+ 585us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:31:39.201119 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:31:39.201214 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:39.201402 (+ 188us) server_negotiation.cc:299] Negotiation successful
0926 16:31:39.201492 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 16:31:39.202509 14429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:39.194412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.3:39807 (local address 127.6.33.1:58447)
0926 16:31:39.195253 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:39.195278 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:39.195444 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:39.196041 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:39.196046 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:39.196059 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 16:31:39.196437 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:39.196447 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:39.197947 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:39.197957 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:39.199388 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:39.199413 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:39.200465 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:39.200470 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:39.200613 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:39.200683 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 16:31:39.201247 (+ 564us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 16:31:39.201268 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 16:31:39.201350 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 16:31:39.201528 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":429,"threads_started":1}
I20260926 16:31:39.237216 13854 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:31:39.237483 13854 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:31:39.240742 13854 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f9211160536747d4bb0d0bceaaae9ffb: Sending DeleteTablet for 3 replicas of tablet 1175a16c0e204c979870968493a33a66
I20260926 16:31:39.241582 13992 tablet_service.cc:1514] Processing DeleteTablet for tablet 1175a16c0e204c979870968493a33a66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:31:39 UTC) from {username='oryx', principal='oryx/127.6.33.62@KRBTEST.COM'} at 127.0.0.1:44908
I20260926 16:31:39.241736 14131 tablet_service.cc:1514] Processing DeleteTablet for tablet 1175a16c0e204c979870968493a33a66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:31:39 UTC) from {username='oryx', principal='oryx/127.6.33.62@KRBTEST.COM'} at 127.0.0.1:58294
I20260926 16:31:39.241987 14271 tablet_service.cc:1514] Processing DeleteTablet for tablet 1175a16c0e204c979870968493a33a66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:31:39 UTC) from {username='oryx', principal='oryx/127.6.33.62@KRBTEST.COM'} at 127.0.0.1:60286
I20260926 16:31:39.259625 14432 tablet_replica.cc:330] T 1175a16c0e204c979870968493a33a66 P ee5950d438ab4f53bab4f057a1712254: stopping tablet replica
I20260926 16:31:39.260053 14432 raft_consensus.cc:2229] T 1175a16c0e204c979870968493a33a66 P ee5950d438ab4f53bab4f057a1712254 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:31:39.260442 14432 raft_consensus.cc:2258] T 1175a16c0e204c979870968493a33a66 P ee5950d438ab4f53bab4f057a1712254 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:31:39.261075 14434 tablet_replica.cc:330] T 1175a16c0e204c979870968493a33a66 P 1eac546f77d24ea08f44d69d57b1cac5: stopping tablet replica
I20260926 16:31:39.261389 14434 raft_consensus.cc:2229] T 1175a16c0e204c979870968493a33a66 P 1eac546f77d24ea08f44d69d57b1cac5 [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:31:39.261833 14434 raft_consensus.cc:2258] T 1175a16c0e204c979870968493a33a66 P 1eac546f77d24ea08f44d69d57b1cac5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:31:39.261870 14432 ts_tablet_manager.cc:1905] T 1175a16c0e204c979870968493a33a66 P ee5950d438ab4f53bab4f057a1712254: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:31:39.265897 14434 ts_tablet_manager.cc:1905] T 1175a16c0e204c979870968493a33a66 P 1eac546f77d24ea08f44d69d57b1cac5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:31:39.267999 14433 tablet_replica.cc:330] T 1175a16c0e204c979870968493a33a66 P 338f775607c644ba8be145b78f82edb7: stopping tablet replica
I20260926 16:31:39.268203 14433 raft_consensus.cc:2229] T 1175a16c0e204c979870968493a33a66 P 338f775607c644ba8be145b78f82edb7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:31:39.268414 14433 raft_consensus.cc:2258] T 1175a16c0e204c979870968493a33a66 P 338f775607c644ba8be145b78f82edb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:31:39.271167 14432 ts_tablet_manager.cc:1918] T 1175a16c0e204c979870968493a33a66 P ee5950d438ab4f53bab4f057a1712254: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 16:31:39.271309 14432 log.cc:1196] T 1175a16c0e204c979870968493a33a66 P ee5950d438ab4f53bab4f057a1712254: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/ts-0/wal/wals/1175a16c0e204c979870968493a33a66
I20260926 16:31:39.271754 14432 ts_tablet_manager.cc:1939] T 1175a16c0e204c979870968493a33a66 P ee5950d438ab4f53bab4f057a1712254: Deleting consensus metadata
I20260926 16:31:39.272156 14433 ts_tablet_manager.cc:1905] T 1175a16c0e204c979870968493a33a66 P 338f775607c644ba8be145b78f82edb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:31:39.272969 13841 catalog_manager.cc:4841] TS ee5950d438ab4f53bab4f057a1712254 (127.6.33.1:32903): tablet 1175a16c0e204c979870968493a33a66 (table test-table [id=8baa762cd4a14e0fa14b9fe4cd16d466]) successfully deleted
I20260926 16:31:39.273838 14434 ts_tablet_manager.cc:1918] T 1175a16c0e204c979870968493a33a66 P 1eac546f77d24ea08f44d69d57b1cac5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 16:31:39.273972 14434 log.cc:1196] T 1175a16c0e204c979870968493a33a66 P 1eac546f77d24ea08f44d69d57b1cac5: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/ts-2/wal/wals/1175a16c0e204c979870968493a33a66
I20260926 16:31:39.274394 14434 ts_tablet_manager.cc:1939] T 1175a16c0e204c979870968493a33a66 P 1eac546f77d24ea08f44d69d57b1cac5: Deleting consensus metadata
I20260926 16:31:39.275520 13840 catalog_manager.cc:4841] TS 1eac546f77d24ea08f44d69d57b1cac5 (127.6.33.3:39807): tablet 1175a16c0e204c979870968493a33a66 (table test-table [id=8baa762cd4a14e0fa14b9fe4cd16d466]) successfully deleted
I20260926 16:31:39.280378 14433 ts_tablet_manager.cc:1918] T 1175a16c0e204c979870968493a33a66 P 338f775607c644ba8be145b78f82edb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 16:31:39.280563 14433 log.cc:1196] T 1175a16c0e204c979870968493a33a66 P 338f775607c644ba8be145b78f82edb7: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/ts-1/wal/wals/1175a16c0e204c979870968493a33a66
I20260926 16:31:39.281015 14433 ts_tablet_manager.cc:1939] T 1175a16c0e204c979870968493a33a66 P 338f775607c644ba8be145b78f82edb7: Deleting consensus metadata
I20260926 16:31:39.282181 13842 catalog_manager.cc:4841] TS 338f775607c644ba8be145b78f82edb7 (127.6.33.2:36057): tablet 1175a16c0e204c979870968493a33a66 (table test-table [id=8baa762cd4a14e0fa14b9fe4cd16d466]) successfully deleted
Sep 26 16:31:39 dist-test-slave-khd3 krb5kdc[13810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
W20260926 16:31:39.327224 14337 messenger.cc:188] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.6.33.3:39807
Sep 26 16:31:39 dist-test-slave-khd3 krb5kdc[13810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:31:39 dist-test-slave-khd3 krb5kdc[13810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 16:31:39.339733 14442 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:31:39.310755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44834 (local address 127.6.33.62:38215)
0926 16:31:39.323560 (+ 12805us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:39.323567 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:39.323588 (+ 21us) server_negotiation.cc:407] Connection header received
0926 16:31:39.323651 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:39.323656 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:39.323729 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:39.323853 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:39.325324 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:39.326083 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:39.327380 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:39.327825 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:39.327970 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:39.339491 (+ 11521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44834: BlockingRecv error: recv got EOF from 127.0.0.1:44834 (error 108)
Metrics: {"server-negotiator.queue_time_us":12664,"thread_start_us":12774,"threads_started":1}
Sep 26 16:31:39 dist-test-slave-khd3 krb5kdc[13810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM
I20260926 16:31:39.360337 14442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:39.344938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44836 (local address 127.6.33.62:38215)
0926 16:31:39.345149 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:39.345159 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:39.345673 (+ 514us) server_negotiation.cc:407] Connection header received
0926 16:31:39.345891 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:39.345896 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:39.345958 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:39.346065 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:39.347453 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:39.348315 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:39.349632 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:39.350066 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:39.350204 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:39.354451 (+ 4247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:39.354458 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:39.354462 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:39.354508 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:39.357473 (+ 2965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:39.358125 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:39.358137 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:39.358139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:39.358357 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:39.358808 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:39.358813 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:39.358816 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:39.359139 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:39.359398 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:39.359767 (+ 369us) server_negotiation.cc:299] Negotiation successful
0926 16:31:39.359997 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 16:31:39.363865 13854 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44836:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:31:39.364514 13854 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:31:39.371452 13992 tablet_service.cc:1467] Processing CreateTablet for tablet 5b987a8552fb4a63adb214fa70f1e7b7 (DEFAULT_TABLE table=test-table [id=35628759bf8a4849868e329bcdb10a1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:31:39.371707 14271 tablet_service.cc:1467] Processing CreateTablet for tablet 5b987a8552fb4a63adb214fa70f1e7b7 (DEFAULT_TABLE table=test-table [id=35628759bf8a4849868e329bcdb10a1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:31:39.371945 13992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b987a8552fb4a63adb214fa70f1e7b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:31:39.372095 14271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b987a8552fb4a63adb214fa70f1e7b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:31:39.377774 14131 tablet_service.cc:1467] Processing CreateTablet for tablet 5b987a8552fb4a63adb214fa70f1e7b7 (DEFAULT_TABLE table=test-table [id=35628759bf8a4849868e329bcdb10a1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:31:39.378202 14131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b987a8552fb4a63adb214fa70f1e7b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:31:39.383870 14449 tablet_bootstrap.cc:492] T 5b987a8552fb4a63adb214fa70f1e7b7 P ee5950d438ab4f53bab4f057a1712254: Bootstrap starting.
I20260926 16:31:39.384516 14448 tablet_bootstrap.cc:492] T 5b987a8552fb4a63adb214fa70f1e7b7 P 1eac546f77d24ea08f44d69d57b1cac5: Bootstrap starting.
I20260926 16:31:39.386147 14449 tablet_bootstrap.cc:654] T 5b987a8552fb4a63adb214fa70f1e7b7 P ee5950d438ab4f53bab4f057a1712254: Neither blocks nor log segments found. Creating new log.
I20260926 16:31:39.386279 14448 tablet_bootstrap.cc:654] T 5b987a8552fb4a63adb214fa70f1e7b7 P 1eac546f77d24ea08f44d69d57b1cac5: Neither blocks nor log segments found. Creating new log.
I20260926 16:31:39.394860 14448 tablet_bootstrap.cc:492] T 5b987a8552fb4a63adb214fa70f1e7b7 P 1eac546f77d24ea08f44d69d57b1cac5: No bootstrap required, opened a new log
I20260926 16:31:39.395046 14448 ts_tablet_manager.cc:1397] T 5b987a8552fb4a63adb214fa70f1e7b7 P 1eac546f77d24ea08f44d69d57b1cac5: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20260926 16:31:39.395273 14449 tablet_bootstrap.cc:492] T 5b987a8552fb4a63adb214fa70f1e7b7 P ee5950d438ab4f53bab4f057a1712254: No bootstrap required, opened a new log
I20260926 16:31:39.395433 14449 ts_tablet_manager.cc:1397] T 5b987a8552fb4a63adb214fa70f1e7b7 P ee5950d438ab4f53bab4f057a1712254: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20260926 16:31:39.395923 14448 raft_consensus.cc:348] T 5b987a8552fb4a63adb214fa70f1e7b7 P 1eac546f77d24ea08f44d69d57b1cac5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5950d438ab4f53bab4f057a1712254" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 32903 } } peers { permanent_uuid: "338f775607c644ba8be145b78f82edb7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36057 } } peers { permanent_uuid: "1eac546f77d24ea08f44d69d57b1cac5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39807 } }
I20260926 16:31:39.396168 14448 raft_consensus.cc:374] T 5b987a8552fb4a63adb214fa70f1e7b7 P 1eac546f77d24ea08f44d69d57b1cac5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:31:39.396279 14448 raft_consensus.cc:729] T 5b987a8552fb4a63adb214fa70f1e7b7 P 1eac546f77d24ea08f44d69d57b1cac5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eac546f77d24ea08f44d69d57b1cac5, State: Initialized, Role: FOLLOWER
I20260926 16:31:39.396214 14449 raft_consensus.cc:348] T 5b987a8552fb4a63adb214fa70f1e7b7 P ee5950d438ab4f53bab4f057a1712254 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5950d438ab4f53bab4f057a1712254" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 32903 } } peers { permanent_uuid: "338f775607c644ba8be145b78f82edb7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36057 } } peers { permanent_uuid: "1eac546f77d24ea08f44d69d57b1cac5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39807 } }
I20260926 16:31:39.396479 14449 raft_consensus.cc:374] T 5b987a8552fb4a63adb214fa70f1e7b7 P ee5950d438ab4f53bab4f057a1712254 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:31:39.396595 14449 raft_consensus.cc:729] T 5b987a8552fb4a63adb214fa70f1e7b7 P ee5950d438ab4f53bab4f057a1712254 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee5950d438ab4f53bab4f057a1712254, State: Initialized, Role: FOLLOWER
I20260926 16:31:39.396623 14448 consensus_queue.cc:260] T 5b987a8552fb4a63adb214fa70f1e7b7 P 1eac546f77d24ea08f44d69d57b1cac5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5950d438ab4f53bab4f057a1712254" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 32903 } } peers { permanent_uuid: "338f775607c644ba8be145b78f82edb7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36057 } } peers { permanent_uuid: "1eac546f77d24ea08f44d69d57b1cac5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39807 } }
I20260926 16:31:39.396915 14449 consensus_queue.cc:260] T 5b987a8552fb4a63adb214fa70f1e7b7 P ee5950d438ab4f53bab4f057a1712254 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5950d438ab4f53bab4f057a1712254" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 32903 } } peers { permanent_uuid: "338f775607c644ba8be145b78f82edb7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36057 } } peers { permanent_uuid: "1eac546f77d24ea08f44d69d57b1cac5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39807 } }
I20260926 16:31:39.397466 14449 ts_tablet_manager.cc:1428] T 5b987a8552fb4a63adb214fa70f1e7b7 P ee5950d438ab4f53bab4f057a1712254: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260926 16:31:39.398012 14448 ts_tablet_manager.cc:1428] T 5b987a8552fb4a63adb214fa70f1e7b7 P 1eac546f77d24ea08f44d69d57b1cac5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 16:31:39.398661 14452 tablet_bootstrap.cc:492] T 5b987a8552fb4a63adb214fa70f1e7b7 P 338f775607c644ba8be145b78f82edb7: Bootstrap starting.
I20260926 16:31:39.401013 14452 tablet_bootstrap.cc:654] T 5b987a8552fb4a63adb214fa70f1e7b7 P 338f775607c644ba8be145b78f82edb7: Neither blocks nor log segments found. Creating new log.
I20260926 16:31:39.403558 14452 tablet_bootstrap.cc:492] T 5b987a8552fb4a63adb214fa70f1e7b7 P 338f775607c644ba8be145b78f82edb7: No bootstrap required, opened a new log
I20260926 16:31:39.403712 14452 ts_tablet_manager.cc:1397] T 5b987a8552fb4a63adb214fa70f1e7b7 P 338f775607c644ba8be145b78f82edb7: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20260926 16:31:39.404368 14452 raft_consensus.cc:348] T 5b987a8552fb4a63adb214fa70f1e7b7 P 338f775607c644ba8be145b78f82edb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5950d438ab4f53bab4f057a1712254" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 32903 } } peers { permanent_uuid: "338f775607c644ba8be145b78f82edb7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36057 } } peers { permanent_uuid: "1eac546f77d24ea08f44d69d57b1cac5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39807 } }
I20260926 16:31:39.404654 14452 raft_consensus.cc:374] T 5b987a8552fb4a63adb214fa70f1e7b7 P 338f775607c644ba8be145b78f82edb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:31:39.404775 14452 raft_consensus.cc:729] T 5b987a8552fb4a63adb214fa70f1e7b7 P 338f775607c644ba8be145b78f82edb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 338f775607c644ba8be145b78f82edb7, State: Initialized, Role: FOLLOWER
I20260926 16:31:39.404999 14452 consensus_queue.cc:260] T 5b987a8552fb4a63adb214fa70f1e7b7 P 338f775607c644ba8be145b78f82edb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5950d438ab4f53bab4f057a1712254" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 32903 } } peers { permanent_uuid: "338f775607c644ba8be145b78f82edb7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36057 } } peers { permanent_uuid: "1eac546f77d24ea08f44d69d57b1cac5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39807 } }
I20260926 16:31:39.405463 14452 ts_tablet_manager.cc:1428] T 5b987a8552fb4a63adb214fa70f1e7b7 P 338f775607c644ba8be145b78f82edb7: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260926 16:31:39.410317 14421 raft_consensus.cc:482] T 5b987a8552fb4a63adb214fa70f1e7b7 P 338f775607c644ba8be145b78f82edb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:31:39.410554 14421 raft_consensus.cc:504] T 5b987a8552fb4a63adb214fa70f1e7b7 P 338f775607c644ba8be145b78f82edb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5950d438ab4f53bab4f057a1712254" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 32903 } } peers { permanent_uuid: "338f775607c644ba8be145b78f82edb7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36057 } } peers { permanent_uuid: "1eac546f77d24ea08f44d69d57b1cac5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39807 } }
I20260926 16:31:39.412160 14421 leader_election.cc:288] T 5b987a8552fb4a63adb214fa70f1e7b7 P 338f775607c644ba8be145b78f82edb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee5950d438ab4f53bab4f057a1712254 (127.6.33.1:32903), 1eac546f77d24ea08f44d69d57b1cac5 (127.6.33.3:39807)
I20260926 16:31:39.419423 14412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:39.412849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:51907 (local address 127.6.33.1:32903)
0926 16:31:39.413053 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:39.413060 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:39.413079 (+ 19us) server_negotiation.cc:407] Connection header received
0926 16:31:39.413297 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:39.413305 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:39.413373 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:39.413429 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:31:39.414061 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:39.415052 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:39.417122 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:39.418459 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:39.418537 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:39.418611 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:39.419002 (+ 391us) server_negotiation.cc:299] Negotiation successful
0926 16:31:39.419137 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 16:31:39.419466 14454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:39.412419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:32903 (local address 127.6.33.2:51907)
0926 16:31:39.412961 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:39.412987 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:39.413113 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:39.413586 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:39.413589 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:39.413608 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:31:39.413900 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:39.413908 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:39.415182 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:39.415189 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:39.416910 (+ 1721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:39.416928 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:39.418737 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:39.418741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:39.418850 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:39.418873 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 16:31:39.418974 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 16:31:39.419105 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":698,"threads_started":1}
I20260926 16:31:39.420130 14012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b987a8552fb4a63adb214fa70f1e7b7" candidate_uuid: "338f775607c644ba8be145b78f82edb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee5950d438ab4f53bab4f057a1712254" is_pre_election: true
I20260926 16:31:39.420390 14012 raft_consensus.cc:2454] T 5b987a8552fb4a63adb214fa70f1e7b7 P ee5950d438ab4f53bab4f057a1712254 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 338f775607c644ba8be145b78f82edb7 in term 0.
I20260926 16:31:39.420925 14084 leader_election.cc:302] T 5b987a8552fb4a63adb214fa70f1e7b7 P 338f775607c644ba8be145b78f82edb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 338f775607c644ba8be145b78f82edb7, ee5950d438ab4f53bab4f057a1712254; no voters:
I20260926 16:31:39.421314 14421 raft_consensus.cc:2781] T 5b987a8552fb4a63adb214fa70f1e7b7 P 338f775607c644ba8be145b78f82edb7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:31:39.421453 14421 raft_consensus.cc:482] T 5b987a8552fb4a63adb214fa70f1e7b7 P 338f775607c644ba8be145b78f82edb7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:31:39.421546 14421 raft_consensus.cc:3037] T 5b987a8552fb4a63adb214fa70f1e7b7 P 338f775607c644ba8be145b78f82edb7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:31:39.424651 14361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:39.412820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:50385 (local address 127.6.33.3:39807)
0926 16:31:39.413020 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:39.413025 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:39.417341 (+ 4316us) server_negotiation.cc:407] Connection header received
0926 16:31:39.417470 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:39.417475 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:39.417554 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:39.417657 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:31:39.418247 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:39.419313 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:39.422115 (+ 2802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:39.423404 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:39.423535 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:39.423648 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:39.424180 (+ 532us) server_negotiation.cc:299] Negotiation successful
0926 16:31:39.424321 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 16:31:39.424719 14421 raft_consensus.cc:504] T 5b987a8552fb4a63adb214fa70f1e7b7 P 338f775607c644ba8be145b78f82edb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5950d438ab4f53bab4f057a1712254" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 32903 } } peers { permanent_uuid: "338f775607c644ba8be145b78f82edb7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36057 } } peers { permanent_uuid: "1eac546f77d24ea08f44d69d57b1cac5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39807 } }
I20260926 16:31:39.425549 14455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:39.412456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.3:39807 (local address 127.6.33.2:50385)
0926 16:31:39.417219 (+ 4763us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:39.417237 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:39.417353 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:39.417684 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:39.417689 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:39.417714 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:31:39.418066 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:39.418076 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:39.419488 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:39.419493 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:39.421939 (+ 2446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:39.421956 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:39.423805 (+ 1849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:39.423813 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:39.423996 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:39.424029 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 16:31:39.425045 (+ 1016us) client_negotiation.cc:240] Negotiation successful
0926 16:31:39.425190 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4637,"thread_start_us":4473,"threads_started":1}
I20260926 16:31:39.425556 14421 leader_election.cc:288] T 5b987a8552fb4a63adb214fa70f1e7b7 P 338f775607c644ba8be145b78f82edb7 [CANDIDATE]: Term 1 election: Requested vote from peers ee5950d438ab4f53bab4f057a1712254 (127.6.33.1:32903), 1eac546f77d24ea08f44d69d57b1cac5 (127.6.33.3:39807)
I20260926 16:31:39.426030 14012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b987a8552fb4a63adb214fa70f1e7b7" candidate_uuid: "338f775607c644ba8be145b78f82edb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee5950d438ab4f53bab4f057a1712254"
I20260926 16:31:39.426321 14012 raft_consensus.cc:3037] T 5b987a8552fb4a63adb214fa70f1e7b7 P ee5950d438ab4f53bab4f057a1712254 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:31:39.426406 14293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b987a8552fb4a63adb214fa70f1e7b7" candidate_uuid: "338f775607c644ba8be145b78f82edb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eac546f77d24ea08f44d69d57b1cac5" is_pre_election: true
I20260926 16:31:39.426589 14295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b987a8552fb4a63adb214fa70f1e7b7" candidate_uuid: "338f775607c644ba8be145b78f82edb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eac546f77d24ea08f44d69d57b1cac5"
I20260926 16:31:39.426717 14293 raft_consensus.cc:2454] T 5b987a8552fb4a63adb214fa70f1e7b7 P 1eac546f77d24ea08f44d69d57b1cac5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 338f775607c644ba8be145b78f82edb7 in term 0.
I20260926 16:31:39.429518 14012 raft_consensus.cc:2454] T 5b987a8552fb4a63adb214fa70f1e7b7 P ee5950d438ab4f53bab4f057a1712254 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 338f775607c644ba8be145b78f82edb7 in term 1.
I20260926 16:31:39.430032 14084 leader_election.cc:302] T 5b987a8552fb4a63adb214fa70f1e7b7 P 338f775607c644ba8be145b78f82edb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 338f775607c644ba8be145b78f82edb7, ee5950d438ab4f53bab4f057a1712254; no voters: 1eac546f77d24ea08f44d69d57b1cac5
I20260926 16:31:39.430493 14421 raft_consensus.cc:2781] T 5b987a8552fb4a63adb214fa70f1e7b7 P 338f775607c644ba8be145b78f82edb7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:31:39.430688 14421 raft_consensus.cc:686] T 5b987a8552fb4a63adb214fa70f1e7b7 P 338f775607c644ba8be145b78f82edb7 [term 1 LEADER]: Becoming Leader. State: Replica: 338f775607c644ba8be145b78f82edb7, State: Running, Role: LEADER
I20260926 16:31:39.431118 14421 consensus_queue.cc:237] T 5b987a8552fb4a63adb214fa70f1e7b7 P 338f775607c644ba8be145b78f82edb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5950d438ab4f53bab4f057a1712254" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 32903 } } peers { permanent_uuid: "338f775607c644ba8be145b78f82edb7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36057 } } peers { permanent_uuid: "1eac546f77d24ea08f44d69d57b1cac5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39807 } }
I20260926 16:31:39.435259 13853 catalog_manager.cc:5495] T 5b987a8552fb4a63adb214fa70f1e7b7 P 338f775607c644ba8be145b78f82edb7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 338f775607c644ba8be145b78f82edb7 (127.6.33.2). New cstate: current_term: 1 leader_uuid: "338f775607c644ba8be145b78f82edb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5950d438ab4f53bab4f057a1712254" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 32903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "338f775607c644ba8be145b78f82edb7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eac546f77d24ea08f44d69d57b1cac5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39807 } health_report { overall_health: UNKNOWN } } }
I20260926 16:31:39.460160 14405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:39.453881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58298 (local address 127.6.33.2:36057)
0926 16:31:39.454063 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:39.454069 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:39.454202 (+ 133us) server_negotiation.cc:407] Connection header received
0926 16:31:39.454377 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:39.454381 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:39.454470 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:39.454598 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:31:39.455023 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:39.455733 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:39.457730 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:39.458142 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:39.458306 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:39.458805 (+ 499us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:31:39.459522 (+ 717us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:31:39.459643 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:39.459836 (+ 193us) server_negotiation.cc:299] Negotiation successful
0926 16:31:39.459902 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 16:31:39.463709 14293 raft_consensus.cc:3037] T 5b987a8552fb4a63adb214fa70f1e7b7 P 1eac546f77d24ea08f44d69d57b1cac5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:31:39.463806 14012 raft_consensus.cc:1261] T 5b987a8552fb4a63adb214fa70f1e7b7 P ee5950d438ab4f53bab4f057a1712254 [term 1 FOLLOWER]: Refusing update from remote peer 338f775607c644ba8be145b78f82edb7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:31:39.464502 14373 consensus_queue.cc:1035] T 5b987a8552fb4a63adb214fa70f1e7b7 P 338f775607c644ba8be145b78f82edb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee5950d438ab4f53bab4f057a1712254" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 32903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:31:39.467378 14293 raft_consensus.cc:1261] T 5b987a8552fb4a63adb214fa70f1e7b7 P 1eac546f77d24ea08f44d69d57b1cac5 [term 1 FOLLOWER]: Refusing update from remote peer 338f775607c644ba8be145b78f82edb7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:31:39.468848 14373 consensus_queue.cc:1035] T 5b987a8552fb4a63adb214fa70f1e7b7 P 338f775607c644ba8be145b78f82edb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eac546f77d24ea08f44d69d57b1cac5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:31:39.484947 13853 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:31:39.485296 13853 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:31:39.490063 13853 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f9211160536747d4bb0d0bceaaae9ffb: Sending DeleteTablet for 3 replicas of tablet 5b987a8552fb4a63adb214fa70f1e7b7
I20260926 16:31:39.490942 13992 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b987a8552fb4a63adb214fa70f1e7b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:31:39 UTC) from {username='oryx', principal='oryx/127.6.33.62@KRBTEST.COM'} at 127.0.0.1:44908
I20260926 16:31:39.491312 14432 tablet_replica.cc:330] T 5b987a8552fb4a63adb214fa70f1e7b7 P ee5950d438ab4f53bab4f057a1712254: stopping tablet replica
I20260926 16:31:39.491554 14432 raft_consensus.cc:2229] T 5b987a8552fb4a63adb214fa70f1e7b7 P ee5950d438ab4f53bab4f057a1712254 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:31:39.491796 14432 raft_consensus.cc:2258] T 5b987a8552fb4a63adb214fa70f1e7b7 P ee5950d438ab4f53bab4f057a1712254 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:31:39.492779 14271 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b987a8552fb4a63adb214fa70f1e7b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:31:39 UTC) from {username='oryx', principal='oryx/127.6.33.62@KRBTEST.COM'} at 127.0.0.1:60286
I20260926 16:31:39.493109 14434 tablet_replica.cc:330] T 5b987a8552fb4a63adb214fa70f1e7b7 P 1eac546f77d24ea08f44d69d57b1cac5: stopping tablet replica
I20260926 16:31:39.493132 14131 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b987a8552fb4a63adb214fa70f1e7b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:31:39 UTC) from {username='oryx', principal='oryx/127.6.33.62@KRBTEST.COM'} at 127.0.0.1:58294
I20260926 16:31:39.493297 14434 raft_consensus.cc:2229] T 5b987a8552fb4a63adb214fa70f1e7b7 P 1eac546f77d24ea08f44d69d57b1cac5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:31:39.493141 13854 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44836:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 16:31:39.493417 14433 tablet_replica.cc:330] T 5b987a8552fb4a63adb214fa70f1e7b7 P 338f775607c644ba8be145b78f82edb7: stopping tablet replica
I20260926 16:31:39.493461 14434 pending_rounds.cc:62] T 5b987a8552fb4a63adb214fa70f1e7b7 P 1eac546f77d24ea08f44d69d57b1cac5: Trying to abort 2 pending ops.
I20260926 16:31:39.493621 14433 raft_consensus.cc:2229] T 5b987a8552fb4a63adb214fa70f1e7b7 P 338f775607c644ba8be145b78f82edb7 [term 1 LEADER]: Raft consensus shutting down.
W20260926 16:31:39.493808 13854 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:31:39.493575 14434 pending_rounds.cc:68] T 5b987a8552fb4a63adb214fa70f1e7b7 P 1eac546f77d24ea08f44d69d57b1cac5: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7333643466596704256 op_type: WRITE_OP write_request { tablet_id: "5b987a8552fb4a63adb214fa70f1e7b7" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\227\351\337\325\006\"8\n\ntest-admin\022*\n 35628759bf8a4849868e329bcdb10a1c\020\001\030\001 \001(\001" signature: "<redacted>""\261u\251\026\257\257\317\030-\342\024k\331\177\357i^b\301?y\306\221\354m6\223\207\361\336\331\203\324\0036\rd\307\207\036\210\332\243[\21310\000I\000\276\316]\256_~\341a\r=\224\301\250\333" signing_key_seq_num: 0 } } request_id { client_id: "b47053c177ba469e91b0b0c43c75346c" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260926 16:31:39.494079 14433 raft_consensus.cc:2258] T 5b987a8552fb4a63adb214fa70f1e7b7 P 338f775607c644ba8be145b78f82edb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:31:39.494331 14434 pending_rounds.cc:68] T 5b987a8552fb4a63adb214fa70f1e7b7 P 1eac546f77d24ea08f44d69d57b1cac5: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7333643466473635840 op_type: NO_OP noop_request { }
I20260926 16:31:39.494494 14434 raft_consensus.cc:2866] T 5b987a8552fb4a63adb214fa70f1e7b7 P 1eac546f77d24ea08f44d69d57b1cac5 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260926 16:31:39.494608 14434 raft_consensus.cc:2258] T 5b987a8552fb4a63adb214fa70f1e7b7 P 1eac546f77d24ea08f44d69d57b1cac5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:31:39.494697 14432 ts_tablet_manager.cc:1905] T 5b987a8552fb4a63adb214fa70f1e7b7 P ee5950d438ab4f53bab4f057a1712254: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:31:39.496016 14433 ts_tablet_manager.cc:1905] T 5b987a8552fb4a63adb214fa70f1e7b7 P 338f775607c644ba8be145b78f82edb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:31:39.501000 13992 tablet_service.cc:1467] Processing CreateTablet for tablet 85930f382b6841d181d6cfc180979331 (DEFAULT_TABLE table=test-table [id=6c8461f35ba944c1bd15c6dda5eb82cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:31:39.501502 13992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85930f382b6841d181d6cfc180979331. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:31:39.501571 14131 tablet_service.cc:1467] Processing CreateTablet for tablet 85930f382b6841d181d6cfc180979331 (DEFAULT_TABLE table=test-table [id=6c8461f35ba944c1bd15c6dda5eb82cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:31:39.501988 14131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85930f382b6841d181d6cfc180979331. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:31:39.505210 14271 tablet_service.cc:1467] Processing CreateTablet for tablet 85930f382b6841d181d6cfc180979331 (DEFAULT_TABLE table=test-table [id=6c8461f35ba944c1bd15c6dda5eb82cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:31:39.505658 14271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85930f382b6841d181d6cfc180979331. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:31:39.507154 14449 tablet_bootstrap.cc:492] T 85930f382b6841d181d6cfc180979331 P ee5950d438ab4f53bab4f057a1712254: Bootstrap starting.
I20260926 16:31:39.509004 14449 tablet_bootstrap.cc:654] T 85930f382b6841d181d6cfc180979331 P ee5950d438ab4f53bab4f057a1712254: Neither blocks nor log segments found. Creating new log.
I20260926 16:31:39.509755 14433 ts_tablet_manager.cc:1918] T 5b987a8552fb4a63adb214fa70f1e7b7 P 338f775607c644ba8be145b78f82edb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:31:39.509899 14433 log.cc:1196] T 5b987a8552fb4a63adb214fa70f1e7b7 P 338f775607c644ba8be145b78f82edb7: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/ts-1/wal/wals/5b987a8552fb4a63adb214fa70f1e7b7
I20260926 16:31:39.510644 14433 ts_tablet_manager.cc:1939] T 5b987a8552fb4a63adb214fa70f1e7b7 P 338f775607c644ba8be145b78f82edb7: Deleting consensus metadata
I20260926 16:31:39.511739 13842 catalog_manager.cc:4841] TS 338f775607c644ba8be145b78f82edb7 (127.6.33.2:36057): tablet 5b987a8552fb4a63adb214fa70f1e7b7 (table test-table [id=35628759bf8a4849868e329bcdb10a1c]) successfully deleted
I20260926 16:31:39.512898 14434 ts_tablet_manager.cc:1905] T 5b987a8552fb4a63adb214fa70f1e7b7 P 1eac546f77d24ea08f44d69d57b1cac5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:31:39.515198 14452 tablet_bootstrap.cc:492] T 85930f382b6841d181d6cfc180979331 P 338f775607c644ba8be145b78f82edb7: Bootstrap starting.
I20260926 16:31:39.516923 14452 tablet_bootstrap.cc:654] T 85930f382b6841d181d6cfc180979331 P 338f775607c644ba8be145b78f82edb7: Neither blocks nor log segments found. Creating new log.
I20260926 16:31:39.517814 14448 tablet_bootstrap.cc:492] T 85930f382b6841d181d6cfc180979331 P 1eac546f77d24ea08f44d69d57b1cac5: Bootstrap starting.
I20260926 16:31:39.519999 14432 ts_tablet_manager.cc:1918] T 5b987a8552fb4a63adb214fa70f1e7b7 P ee5950d438ab4f53bab4f057a1712254: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:31:39.520058 14448 tablet_bootstrap.cc:654] T 85930f382b6841d181d6cfc180979331 P 1eac546f77d24ea08f44d69d57b1cac5: Neither blocks nor log segments found. Creating new log.
I20260926 16:31:39.522564 14449 tablet_bootstrap.cc:492] T 85930f382b6841d181d6cfc180979331 P ee5950d438ab4f53bab4f057a1712254: No bootstrap required, opened a new log
I20260926 16:31:39.522586 14432 log.cc:1196] T 5b987a8552fb4a63adb214fa70f1e7b7 P ee5950d438ab4f53bab4f057a1712254: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/ts-0/wal/wals/5b987a8552fb4a63adb214fa70f1e7b7
I20260926 16:31:39.522749 14449 ts_tablet_manager.cc:1397] T 85930f382b6841d181d6cfc180979331 P ee5950d438ab4f53bab4f057a1712254: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.004s
I20260926 16:31:39.523422 14434 ts_tablet_manager.cc:1918] T 5b987a8552fb4a63adb214fa70f1e7b7 P 1eac546f77d24ea08f44d69d57b1cac5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:31:39.523603 14434 log.cc:1196] T 5b987a8552fb4a63adb214fa70f1e7b7 P 1eac546f77d24ea08f44d69d57b1cac5: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/ts-2/wal/wals/5b987a8552fb4a63adb214fa70f1e7b7
I20260926 16:31:39.523804 14449 raft_consensus.cc:348] T 85930f382b6841d181d6cfc180979331 P ee5950d438ab4f53bab4f057a1712254 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5950d438ab4f53bab4f057a1712254" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 32903 } } peers { permanent_uuid: "338f775607c644ba8be145b78f82edb7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36057 } } peers { permanent_uuid: "1eac546f77d24ea08f44d69d57b1cac5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39807 } }
I20260926 16:31:39.524107 14434 ts_tablet_manager.cc:1939] T 5b987a8552fb4a63adb214fa70f1e7b7 P 1eac546f77d24ea08f44d69d57b1cac5: Deleting consensus metadata
I20260926 16:31:39.524144 14449 raft_consensus.cc:374] T 85930f382b6841d181d6cfc180979331 P ee5950d438ab4f53bab4f057a1712254 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:31:39.524331 14449 raft_consensus.cc:729] T 85930f382b6841d181d6cfc180979331 P ee5950d438ab4f53bab4f057a1712254 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee5950d438ab4f53bab4f057a1712254, State: Initialized, Role: FOLLOWER
I20260926 16:31:39.524696 14449 consensus_queue.cc:260] T 85930f382b6841d181d6cfc180979331 P ee5950d438ab4f53bab4f057a1712254 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5950d438ab4f53bab4f057a1712254" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 32903 } } peers { permanent_uuid: "338f775607c644ba8be145b78f82edb7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36057 } } peers { permanent_uuid: "1eac546f77d24ea08f44d69d57b1cac5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39807 } }
I20260926 16:31:39.525405 14449 ts_tablet_manager.cc:1428] T 85930f382b6841d181d6cfc180979331 P ee5950d438ab4f53bab4f057a1712254: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 16:31:39.525441 13840 catalog_manager.cc:4841] TS 1eac546f77d24ea08f44d69d57b1cac5 (127.6.33.3:39807): tablet 5b987a8552fb4a63adb214fa70f1e7b7 (table test-table [id=35628759bf8a4849868e329bcdb10a1c]) successfully deleted
I20260926 16:31:39.526127 14432 ts_tablet_manager.cc:1939] T 5b987a8552fb4a63adb214fa70f1e7b7 P ee5950d438ab4f53bab4f057a1712254: Deleting consensus metadata
I20260926 16:31:39.527212 13841 catalog_manager.cc:4841] TS ee5950d438ab4f53bab4f057a1712254 (127.6.33.1:32903): tablet 5b987a8552fb4a63adb214fa70f1e7b7 (table test-table [id=35628759bf8a4849868e329bcdb10a1c]) successfully deleted
I20260926 16:31:39.536495 14448 tablet_bootstrap.cc:492] T 85930f382b6841d181d6cfc180979331 P 1eac546f77d24ea08f44d69d57b1cac5: No bootstrap required, opened a new log
I20260926 16:31:39.536867 14448 ts_tablet_manager.cc:1397] T 85930f382b6841d181d6cfc180979331 P 1eac546f77d24ea08f44d69d57b1cac5: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.001s
I20260926 16:31:39.537735 14448 raft_consensus.cc:348] T 85930f382b6841d181d6cfc180979331 P 1eac546f77d24ea08f44d69d57b1cac5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5950d438ab4f53bab4f057a1712254" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 32903 } } peers { permanent_uuid: "338f775607c644ba8be145b78f82edb7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36057 } } peers { permanent_uuid: "1eac546f77d24ea08f44d69d57b1cac5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39807 } }
I20260926 16:31:39.538116 14448 raft_consensus.cc:374] T 85930f382b6841d181d6cfc180979331 P 1eac546f77d24ea08f44d69d57b1cac5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:31:39.538302 14448 raft_consensus.cc:729] T 85930f382b6841d181d6cfc180979331 P 1eac546f77d24ea08f44d69d57b1cac5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eac546f77d24ea08f44d69d57b1cac5, State: Initialized, Role: FOLLOWER
I20260926 16:31:39.538725 14448 consensus_queue.cc:260] T 85930f382b6841d181d6cfc180979331 P 1eac546f77d24ea08f44d69d57b1cac5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5950d438ab4f53bab4f057a1712254" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 32903 } } peers { permanent_uuid: "338f775607c644ba8be145b78f82edb7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36057 } } peers { permanent_uuid: "1eac546f77d24ea08f44d69d57b1cac5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39807 } }
I20260926 16:31:39.542642 14448 ts_tablet_manager.cc:1428] T 85930f382b6841d181d6cfc180979331 P 1eac546f77d24ea08f44d69d57b1cac5: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260926 16:31:39.547910 14452 tablet_bootstrap.cc:492] T 85930f382b6841d181d6cfc180979331 P 338f775607c644ba8be145b78f82edb7: No bootstrap required, opened a new log
I20260926 16:31:39.548167 14452 ts_tablet_manager.cc:1397] T 85930f382b6841d181d6cfc180979331 P 338f775607c644ba8be145b78f82edb7: Time spent bootstrapping tablet: real 0.033s user 0.005s sys 0.019s
I20260926 16:31:39.548908 14452 raft_consensus.cc:348] T 85930f382b6841d181d6cfc180979331 P 338f775607c644ba8be145b78f82edb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5950d438ab4f53bab4f057a1712254" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 32903 } } peers { permanent_uuid: "338f775607c644ba8be145b78f82edb7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36057 } } peers { permanent_uuid: "1eac546f77d24ea08f44d69d57b1cac5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39807 } }
I20260926 16:31:39.549142 14452 raft_consensus.cc:374] T 85930f382b6841d181d6cfc180979331 P 338f775607c644ba8be145b78f82edb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:31:39.549252 14452 raft_consensus.cc:729] T 85930f382b6841d181d6cfc180979331 P 338f775607c644ba8be145b78f82edb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 338f775607c644ba8be145b78f82edb7, State: Initialized, Role: FOLLOWER
I20260926 16:31:39.549521 14452 consensus_queue.cc:260] T 85930f382b6841d181d6cfc180979331 P 338f775607c644ba8be145b78f82edb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5950d438ab4f53bab4f057a1712254" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 32903 } } peers { permanent_uuid: "338f775607c644ba8be145b78f82edb7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36057 } } peers { permanent_uuid: "1eac546f77d24ea08f44d69d57b1cac5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39807 } }
I20260926 16:31:39.549996 14452 ts_tablet_manager.cc:1428] T 85930f382b6841d181d6cfc180979331 P 338f775607c644ba8be145b78f82edb7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 16:31:39.722193 14427 raft_consensus.cc:482] T 85930f382b6841d181d6cfc180979331 P ee5950d438ab4f53bab4f057a1712254 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:31:39.722476 14427 raft_consensus.cc:504] T 85930f382b6841d181d6cfc180979331 P ee5950d438ab4f53bab4f057a1712254 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5950d438ab4f53bab4f057a1712254" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 32903 } } peers { permanent_uuid: "338f775607c644ba8be145b78f82edb7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36057 } } peers { permanent_uuid: "1eac546f77d24ea08f44d69d57b1cac5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39807 } }
I20260926 16:31:39.723157 14427 leader_election.cc:288] T 85930f382b6841d181d6cfc180979331 P ee5950d438ab4f53bab4f057a1712254 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 338f775607c644ba8be145b78f82edb7 (127.6.33.2:36057), 1eac546f77d24ea08f44d69d57b1cac5 (127.6.33.3:39807)
I20260926 16:31:39.723737 14156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85930f382b6841d181d6cfc180979331" candidate_uuid: "ee5950d438ab4f53bab4f057a1712254" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "338f775607c644ba8be145b78f82edb7" is_pre_election: true
I20260926 16:31:39.723755 14293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85930f382b6841d181d6cfc180979331" candidate_uuid: "ee5950d438ab4f53bab4f057a1712254" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eac546f77d24ea08f44d69d57b1cac5" is_pre_election: true
I20260926 16:31:39.724002 14293 raft_consensus.cc:2454] T 85930f382b6841d181d6cfc180979331 P 1eac546f77d24ea08f44d69d57b1cac5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee5950d438ab4f53bab4f057a1712254 in term 0.
I20260926 16:31:39.724007 14156 raft_consensus.cc:2454] T 85930f382b6841d181d6cfc180979331 P 338f775607c644ba8be145b78f82edb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee5950d438ab4f53bab4f057a1712254 in term 0.
I20260926 16:31:39.724471 13943 leader_election.cc:302] T 85930f382b6841d181d6cfc180979331 P ee5950d438ab4f53bab4f057a1712254 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eac546f77d24ea08f44d69d57b1cac5, ee5950d438ab4f53bab4f057a1712254; no voters:
I20260926 16:31:39.724817 14427 raft_consensus.cc:2781] T 85930f382b6841d181d6cfc180979331 P ee5950d438ab4f53bab4f057a1712254 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:31:39.724946 14427 raft_consensus.cc:482] T 85930f382b6841d181d6cfc180979331 P ee5950d438ab4f53bab4f057a1712254 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:31:39.725059 14427 raft_consensus.cc:3037] T 85930f382b6841d181d6cfc180979331 P ee5950d438ab4f53bab4f057a1712254 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:31:39.727511 14427 raft_consensus.cc:504] T 85930f382b6841d181d6cfc180979331 P ee5950d438ab4f53bab4f057a1712254 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5950d438ab4f53bab4f057a1712254" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 32903 } } peers { permanent_uuid: "338f775607c644ba8be145b78f82edb7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36057 } } peers { permanent_uuid: "1eac546f77d24ea08f44d69d57b1cac5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39807 } }
I20260926 16:31:39.728166 14427 leader_election.cc:288] T 85930f382b6841d181d6cfc180979331 P ee5950d438ab4f53bab4f057a1712254 [CANDIDATE]: Term 1 election: Requested vote from peers 338f775607c644ba8be145b78f82edb7 (127.6.33.2:36057), 1eac546f77d24ea08f44d69d57b1cac5 (127.6.33.3:39807)
I20260926 16:31:39.728704 14293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85930f382b6841d181d6cfc180979331" candidate_uuid: "ee5950d438ab4f53bab4f057a1712254" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eac546f77d24ea08f44d69d57b1cac5"
I20260926 16:31:39.728739 14156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85930f382b6841d181d6cfc180979331" candidate_uuid: "ee5950d438ab4f53bab4f057a1712254" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "338f775607c644ba8be145b78f82edb7"
I20260926 16:31:39.728962 14293 raft_consensus.cc:3037] T 85930f382b6841d181d6cfc180979331 P 1eac546f77d24ea08f44d69d57b1cac5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:31:39.729013 14156 raft_consensus.cc:3037] T 85930f382b6841d181d6cfc180979331 P 338f775607c644ba8be145b78f82edb7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:31:39.731443 14156 raft_consensus.cc:2454] T 85930f382b6841d181d6cfc180979331 P 338f775607c644ba8be145b78f82edb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee5950d438ab4f53bab4f057a1712254 in term 1.
I20260926 16:31:39.731899 14293 raft_consensus.cc:2454] T 85930f382b6841d181d6cfc180979331 P 1eac546f77d24ea08f44d69d57b1cac5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee5950d438ab4f53bab4f057a1712254 in term 1.
I20260926 16:31:39.731925 13945 leader_election.cc:302] T 85930f382b6841d181d6cfc180979331 P ee5950d438ab4f53bab4f057a1712254 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 338f775607c644ba8be145b78f82edb7, ee5950d438ab4f53bab4f057a1712254; no voters:
I20260926 16:31:39.732618 14427 raft_consensus.cc:2781] T 85930f382b6841d181d6cfc180979331 P ee5950d438ab4f53bab4f057a1712254 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:31:39.733124 14427 raft_consensus.cc:686] T 85930f382b6841d181d6cfc180979331 P ee5950d438ab4f53bab4f057a1712254 [term 1 LEADER]: Becoming Leader. State: Replica: ee5950d438ab4f53bab4f057a1712254, State: Running, Role: LEADER
I20260926 16:31:39.733445 14427 consensus_queue.cc:237] T 85930f382b6841d181d6cfc180979331 P ee5950d438ab4f53bab4f057a1712254 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5950d438ab4f53bab4f057a1712254" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 32903 } } peers { permanent_uuid: "338f775607c644ba8be145b78f82edb7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36057 } } peers { permanent_uuid: "1eac546f77d24ea08f44d69d57b1cac5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39807 } }
I20260926 16:31:39.736259 13854 catalog_manager.cc:5495] T 85930f382b6841d181d6cfc180979331 P ee5950d438ab4f53bab4f057a1712254 reported cstate change: term changed from 0 to 1, leader changed from <none> to ee5950d438ab4f53bab4f057a1712254 (127.6.33.1). New cstate: current_term: 1 leader_uuid: "ee5950d438ab4f53bab4f057a1712254" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5950d438ab4f53bab4f057a1712254" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 32903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "338f775607c644ba8be145b78f82edb7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eac546f77d24ea08f44d69d57b1cac5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39807 } health_report { overall_health: UNKNOWN } } }
I20260926 16:31:39.781397 14412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:39.775139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44930 (local address 127.6.33.1:32903)
0926 16:31:39.775314 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:39.775320 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:39.775338 (+ 18us) server_negotiation.cc:407] Connection header received
0926 16:31:39.775399 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:39.775405 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:39.775484 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:39.775579 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:31:39.776171 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:39.777295 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:39.779233 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:39.779872 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:39.779991 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:39.780513 (+ 522us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:31:39.780671 (+ 158us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:31:39.780774 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:39.781001 (+ 227us) server_negotiation.cc:299] Negotiation successful
0926 16:31:39.781093 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 16:31:39.803566 14412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:39.791272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:57173 (local address 127.6.33.1:32903)
0926 16:31:39.791452 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:39.791458 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:39.797479 (+ 6021us) server_negotiation.cc:407] Connection header received
0926 16:31:39.797688 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:39.797693 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:39.797770 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:39.797864 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:31:39.798357 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:39.799425 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:39.801383 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:39.802045 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:39.802165 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:39.802702 (+ 537us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:31:39.802836 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:31:39.802925 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:39.803095 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 16:31:39.803188 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 16:31:39.803679 14469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:39.791207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:32903 (local address 127.6.33.1:57173)
0926 16:31:39.797329 (+ 6122us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:39.797353 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:39.797547 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:39.797918 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:39.797923 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:39.797940 (+ 17us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 16:31:39.798183 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:39.798199 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:39.799614 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:39.799622 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:39.801203 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:39.801218 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:39.802210 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:39.802217 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:39.802413 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:39.802467 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 16:31:39.802943 (+ 476us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 16:31:39.802959 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 16:31:39.803052 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 16:31:39.803198 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5917,"thread_start_us":5810,"threads_started":1}
I20260926 16:31:39.812672 14295 raft_consensus.cc:1261] T 85930f382b6841d181d6cfc180979331 P 1eac546f77d24ea08f44d69d57b1cac5 [term 1 FOLLOWER]: Refusing update from remote peer ee5950d438ab4f53bab4f057a1712254: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:31:39.813733 14470 consensus_queue.cc:1035] T 85930f382b6841d181d6cfc180979331 P ee5950d438ab4f53bab4f057a1712254 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eac546f77d24ea08f44d69d57b1cac5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:31:39.814253 14156 raft_consensus.cc:1261] T 85930f382b6841d181d6cfc180979331 P 338f775607c644ba8be145b78f82edb7 [term 1 FOLLOWER]: Refusing update from remote peer ee5950d438ab4f53bab4f057a1712254: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:31:39.814814 14464 consensus_queue.cc:1035] T 85930f382b6841d181d6cfc180979331 P ee5950d438ab4f53bab4f057a1712254 [LEADER]: Connected to new peer: Peer: permanent_uuid: "338f775607c644ba8be145b78f82edb7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 36057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:31:39.824676 14414 mvcc.cc:205] Tried to move back new op lower bound from 7333643468027691008 to 7333643467710861312. Current Snapshot: MvccSnapshot[applied={T|T < 7333643468027691008}]
I20260926 16:31:39.897948 13853 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:31:39.898286 13853 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:31:39.901916 13853 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f9211160536747d4bb0d0bceaaae9ffb: Sending DeleteTablet for 3 replicas of tablet 85930f382b6841d181d6cfc180979331
I20260926 16:31:39.902846 13991 tablet_service.cc:1514] Processing DeleteTablet for tablet 85930f382b6841d181d6cfc180979331 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:31:39 UTC) from {username='oryx', principal='oryx/127.6.33.62@KRBTEST.COM'} at 127.0.0.1:44908
I20260926 16:31:39.903184 14432 tablet_replica.cc:330] T 85930f382b6841d181d6cfc180979331 P ee5950d438ab4f53bab4f057a1712254: stopping tablet replica
I20260926 16:31:39.903278 14271 tablet_service.cc:1514] Processing DeleteTablet for tablet 85930f382b6841d181d6cfc180979331 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:31:39 UTC) from {username='oryx', principal='oryx/127.6.33.62@KRBTEST.COM'} at 127.0.0.1:60286
I20260926 16:31:39.903426 14432 raft_consensus.cc:2229] T 85930f382b6841d181d6cfc180979331 P ee5950d438ab4f53bab4f057a1712254 [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:31:39.903632 14434 tablet_replica.cc:330] T 85930f382b6841d181d6cfc180979331 P 1eac546f77d24ea08f44d69d57b1cac5: stopping tablet replica
I20260926 16:31:39.903836 14434 raft_consensus.cc:2229] T 85930f382b6841d181d6cfc180979331 P 1eac546f77d24ea08f44d69d57b1cac5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:31:39.904011 14432 raft_consensus.cc:2258] T 85930f382b6841d181d6cfc180979331 P ee5950d438ab4f53bab4f057a1712254 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:31:39.904064 14434 raft_consensus.cc:2258] T 85930f382b6841d181d6cfc180979331 P 1eac546f77d24ea08f44d69d57b1cac5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:31:39.904798 14131 tablet_service.cc:1514] Processing DeleteTablet for tablet 85930f382b6841d181d6cfc180979331 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:31:39 UTC) from {username='oryx', principal='oryx/127.6.33.62@KRBTEST.COM'} at 127.0.0.1:58294
I20260926 16:31:39.905117 14433 tablet_replica.cc:330] T 85930f382b6841d181d6cfc180979331 P 338f775607c644ba8be145b78f82edb7: stopping tablet replica
I20260926 16:31:39.905263 14432 ts_tablet_manager.cc:1905] T 85930f382b6841d181d6cfc180979331 P ee5950d438ab4f53bab4f057a1712254: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:31:39.905282 14433 raft_consensus.cc:2229] T 85930f382b6841d181d6cfc180979331 P 338f775607c644ba8be145b78f82edb7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:31:39.905468 14434 ts_tablet_manager.cc:1905] T 85930f382b6841d181d6cfc180979331 P 1eac546f77d24ea08f44d69d57b1cac5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:31:39.905486 14433 raft_consensus.cc:2258] T 85930f382b6841d181d6cfc180979331 P 338f775607c644ba8be145b78f82edb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:31:39.906634 14433 ts_tablet_manager.cc:1905] T 85930f382b6841d181d6cfc180979331 P 338f775607c644ba8be145b78f82edb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:31:39.912190 14434 ts_tablet_manager.cc:1918] T 85930f382b6841d181d6cfc180979331 P 1eac546f77d24ea08f44d69d57b1cac5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 16:31:39.912398 14434 log.cc:1196] T 85930f382b6841d181d6cfc180979331 P 1eac546f77d24ea08f44d69d57b1cac5: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/ts-2/wal/wals/85930f382b6841d181d6cfc180979331
I20260926 16:31:39.912854 14434 ts_tablet_manager.cc:1939] T 85930f382b6841d181d6cfc180979331 P 1eac546f77d24ea08f44d69d57b1cac5: Deleting consensus metadata
I20260926 16:31:39.913182 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 13926
I20260926 16:31:39.913967 13840 catalog_manager.cc:4841] TS 1eac546f77d24ea08f44d69d57b1cac5 (127.6.33.3:39807): tablet 85930f382b6841d181d6cfc180979331 (table test-table [id=6c8461f35ba944c1bd15c6dda5eb82cb]) successfully deleted
I20260926 16:31:39.914448 14433 ts_tablet_manager.cc:1918] T 85930f382b6841d181d6cfc180979331 P 338f775607c644ba8be145b78f82edb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 16:31:39.914587 14433 log.cc:1196] T 85930f382b6841d181d6cfc180979331 P 338f775607c644ba8be145b78f82edb7: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/ts-1/wal/wals/85930f382b6841d181d6cfc180979331
I20260926 16:31:39.915011 14433 ts_tablet_manager.cc:1939] T 85930f382b6841d181d6cfc180979331 P 338f775607c644ba8be145b78f82edb7: Deleting consensus metadata
I20260926 16:31:39.916134 13842 catalog_manager.cc:4841] TS 338f775607c644ba8be145b78f82edb7 (127.6.33.2:36057): tablet 85930f382b6841d181d6cfc180979331 (table test-table [id=6c8461f35ba944c1bd15c6dda5eb82cb]) successfully deleted
I20260926 16:31:39.920660 14432 ts_tablet_manager.cc:1918] T 85930f382b6841d181d6cfc180979331 P ee5950d438ab4f53bab4f057a1712254: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 16:31:39.921175 14432 log.cc:1196] T 85930f382b6841d181d6cfc180979331 P ee5950d438ab4f53bab4f057a1712254: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440233850748-6276-0/minicluster-data/ts-0/wal/wals/85930f382b6841d181d6cfc180979331
I20260926 16:31:39.921702 14432 ts_tablet_manager.cc:1939] T 85930f382b6841d181d6cfc180979331 P ee5950d438ab4f53bab4f057a1712254: Deleting consensus metadata
I20260926 16:31:39.923019 13841 catalog_manager.cc:4841] TS ee5950d438ab4f53bab4f057a1712254 (127.6.33.1:32903): tablet 85930f382b6841d181d6cfc180979331 (table test-table [id=6c8461f35ba944c1bd15c6dda5eb82cb]) successfully deleted
Sep 26 16:31:39 dist-test-slave-khd3 krb5kdc[13810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.33.1@KRBTEST.COM
I20260926 16:31:39.941437 14412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:39.914554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44938 (local address 127.6.33.1:32903)
0926 16:31:39.914739 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:39.914746 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:39.921105 (+ 6359us) server_negotiation.cc:407] Connection header received
0926 16:31:39.921522 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:39.921528 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:39.921635 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:39.921724 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:39.923639 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:39.924827 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:39.926799 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:39.927524 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:39.927668 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:39.933264 (+ 5596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:39.933288 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:39.933316 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:39.933380 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:39.937758 (+ 4378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:39.938809 (+ 1051us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:39.938820 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:39.938824 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:39.939000 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:39.939459 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:39.939464 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:39.939468 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:39.939938 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:39.940242 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:39.940623 (+ 381us) server_negotiation.cc:299] Negotiation successful
0926 16:31:39.940884 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 16:31:39.942219 14057 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:31:40.073918 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 13926
I20260926 16:31:40.097476 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 14066
Sep 26 16:31:40 dist-test-slave-khd3 krb5kdc[13810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.33.2@KRBTEST.COM
I20260926 16:31:40.112730 14478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:40.098246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58300 (local address 127.6.33.2:36057)
0926 16:31:40.098732 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:40.098736 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:40.098750 (+ 14us) server_negotiation.cc:407] Connection header received
0926 16:31:40.098800 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:40.098804 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:40.098865 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:40.098981 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:40.100283 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:40.101113 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:40.102379 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:40.102796 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:40.102934 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:40.107124 (+ 4190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:40.107139 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:40.107155 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:40.107190 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:40.110152 (+ 2962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:40.110886 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:40.110894 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:40.110902 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:40.111038 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:40.111412 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:40.111416 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:40.111418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:40.111763 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:40.112165 (+ 402us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:40.112301 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 16:31:40.112468 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":253,"threads_started":1}
I20260926 16:31:40.113451 14196 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:31:40.236333 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 14066
I20260926 16:31:40.258580 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 14205
Sep 26 16:31:40 dist-test-slave-khd3 krb5kdc[13810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.33.3@KRBTEST.COM
I20260926 16:31:40.274236 14480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:40.259538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60304 (local address 127.6.33.3:39807)
0926 16:31:40.260026 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:40.260031 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:40.260047 (+ 16us) server_negotiation.cc:407] Connection header received
0926 16:31:40.260100 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:40.260104 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:40.260161 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:40.260254 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:40.261550 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:40.262231 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:40.263534 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:40.263974 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:40.264090 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:40.268533 (+ 4443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:40.268547 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:40.268563 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:40.268607 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:40.271720 (+ 3113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:40.272459 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:40.272466 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:40.272469 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:40.272592 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:40.272986 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:40.272989 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:40.272991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:40.273337 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:40.273571 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:40.273796 (+ 225us) server_negotiation.cc:299] Negotiation successful
0926 16:31:40.273961 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":303,"threads_started":1}
I20260926 16:31:40.274885 14336 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:31:40.402945 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 14205
I20260926 16:31:40.425335 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 13823
I20260926 16:31:40.426705 13895 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:31:40.546217 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 13823
2026-09-26T16:31:40Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5705 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440233850748-6276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:31:40 dist-test-slave-khd3 krb5kdc[14485](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:31:40 dist-test-slave-khd3 krb5kdc[14485](info): set up 2 sockets
Sep 26 16:31:40 dist-test-slave-khd3 krb5kdc[14485](info): commencing operation
krb5kdc: starting...
W20260926 16:31:41.666980 6276 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.003s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Sep 26 16:31:41 dist-test-slave-khd3 krb5kdc[14485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440301, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:31:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:31:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab.
Entry for principal kudu/127.6.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab.
I20260926 16:31:41.904497 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440233850748-6276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440233850748-6276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440233850748-6276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:40655
--webserver_interface=127.6.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:40585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.33.62:40655
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:31:42.252826 14498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:31:42.253196 14498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:31:42.253367 14498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:31:42.260874 14498 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:31:42.260988 14498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:31:42.261082 14498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:31:42.261178 14498 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:31:42.261273 14498 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:31:42.271384 14498 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:40585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440233850748-6276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440233850748-6276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.33.62:40655
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:40655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440233850748-6276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:31:42.272724 14498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:31:42.274197 14498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:31:42.286499 14504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:42.286625 14503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:42.286597 14506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:31:42.287333 14498 server_base.cc:1034] running on GCE node
I20260926 16:31:42.288180 14498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:31:42.289702 14498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:31:42.290920 14498 hybrid_clock.cc:648] HybridClock initialized: now 1790440302290871 us; error 37 us; skew 500 ppm
I20260926 16:31:42.291309 14498 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-26T16:31:42Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1813 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:31:42 dist-test-slave-khd3 krb5kdc[14512](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:31:42 dist-test-slave-khd3 krb5kdc[14512](info): set up 2 sockets
Sep 26 16:31:42 dist-test-slave-khd3 krb5kdc[14512](info): commencing operation
krb5kdc: starting...
W20260926 16:31:43.462735 6276 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.001s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Sep 26 16:31:43 dist-test-slave-khd3 krb5kdc[14512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:31:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:31:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab.
Entry for principal kudu/127.6.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab.
I20260926 16:31:43.690431 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:36267
--webserver_interface=127.6.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:45713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.33.62:36267
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:31:44.013448 14525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:31:44.013737 14525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:31:44.013892 14525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:31:44.021245 14525 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:31:44.021368 14525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:31:44.021507 14525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:31:44.021610 14525 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:31:44.021651 14525 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:31:44.031526 14525 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:45713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.33.62:36267
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:36267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:31:44.032977 14525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:31:44.034364 14525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:31:44.046152 14531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:44.046200 14533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:44.046101 14530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:31:44.047194 14525 server_base.cc:1034] running on GCE node
I20260926 16:31:44.048031 14525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:31:44.049542 14525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:31:44.050777 14525 hybrid_clock.cc:648] HybridClock initialized: now 1790440304050719 us; error 30 us; skew 500 ppm
I20260926 16:31:44.051173 14525 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:31:44 dist-test-slave-khd3 krb5kdc[14512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440304, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:31:44.059072 14525 init.cc:377] Logged in from keytab as kudu/127.6.33.62@KRBTEST.COM (short username kudu)
I20260926 16:31:44.061058 14525 webserver.cc:460] Webserver started at http://127.6.33.62:32935/ using document root <none> and password file <none>
I20260926 16:31:44.061803 14525 fs_manager.cc:359] Metadata directory not provided
I20260926 16:31:44.061949 14525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:31:44.062291 14525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:31:44.065716 14525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/master-0/data/instance:
uuid: "48ba2211021f4aaba9a6ae3342963827"
format_stamp: "Formatted at 2026-09-26 16:31:44 on dist-test-slave-khd3"
server_key: "7d520954569ea777ade954a884151c6c"
server_key_iv: "9316669e46b1e140a39cef513012a121"
server_key_version: "encryptionkey@0"
I20260926 16:31:44.066576 14525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/master-0/wal/instance:
uuid: "48ba2211021f4aaba9a6ae3342963827"
format_stamp: "Formatted at 2026-09-26 16:31:44 on dist-test-slave-khd3"
server_key: "7d520954569ea777ade954a884151c6c"
server_key_iv: "9316669e46b1e140a39cef513012a121"
server_key_version: "encryptionkey@0"
I20260926 16:31:44.071863 14525 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 16:31:44.075480 14540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:44.076949 14525 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:31:44.077142 14525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/master-0/wal
uuid: "48ba2211021f4aaba9a6ae3342963827"
format_stamp: "Formatted at 2026-09-26 16:31:44 on dist-test-slave-khd3"
server_key: "7d520954569ea777ade954a884151c6c"
server_key_iv: "9316669e46b1e140a39cef513012a121"
server_key_version: "encryptionkey@0"
I20260926 16:31:44.077351 14525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:31:44.104384 14525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:31:44.108598 14525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:31:44.108915 14525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:31:44.127199 14525 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.62:36267
I20260926 16:31:44.127230 14591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.62:36267 every 8 connection(s)
I20260926 16:31:44.128572 14525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
I20260926 16:31:44.131875 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 14525
I20260926 16:31:44.132109 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/master-0/wal/instance
I20260926 16:31:44.132598 14592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:31:44.132943 6276 external_mini_cluster.cc:1428] Setting key 5778237e7cb48d5d87c37e82ae3f3646
I20260926 16:31:44.147871 14592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48ba2211021f4aaba9a6ae3342963827: Bootstrap starting.
Sep 26 16:31:44 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:31:44.151243 14592 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48ba2211021f4aaba9a6ae3342963827: Neither blocks nor log segments found. Creating new log.
I20260926 16:31:44.152416 14592 log.cc:824] T 00000000000000000000000000000000 P 48ba2211021f4aaba9a6ae3342963827: Log is configured to *not* fsync() on all Append() calls
I20260926 16:31:44.155457 14592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48ba2211021f4aaba9a6ae3342963827: No bootstrap required, opened a new log
I20260926 16:31:44.158617 14595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:44.135147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52746 (local address 127.6.33.62:36267)
0926 16:31:44.136063 (+ 916us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:44.136077 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:44.136116 (+ 39us) server_negotiation.cc:407] Connection header received
0926 16:31:44.138270 (+ 2154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:44.138300 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:44.138849 (+ 549us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:44.139322 (+ 473us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:44.141998 (+ 2676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:44.143952 (+ 1954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:44.145927 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:44.146682 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:44.146889 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:44.150969 (+ 4080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:44.150978 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:44.150992 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:44.151033 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:44.154617 (+ 3584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:44.155326 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:44.155334 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:44.155342 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:44.155516 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:44.155951 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:44.155954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:44.155956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:44.156476 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:44.156680 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:44.157176 (+ 496us) server_negotiation.cc:299] Negotiation successful
0926 16:31:44.157533 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":709,"thread_start_us":485,"threads_started":1}
I20260926 16:31:44.163452 14592 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48ba2211021f4aaba9a6ae3342963827 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48ba2211021f4aaba9a6ae3342963827" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 36267 } }
I20260926 16:31:44.163811 14592 raft_consensus.cc:374] T 00000000000000000000000000000000 P 48ba2211021f4aaba9a6ae3342963827 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:31:44.163926 14592 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48ba2211021f4aaba9a6ae3342963827 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48ba2211021f4aaba9a6ae3342963827, State: Initialized, Role: FOLLOWER
I20260926 16:31:44.164465 14592 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48ba2211021f4aaba9a6ae3342963827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48ba2211021f4aaba9a6ae3342963827" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 36267 } }
I20260926 16:31:44.164705 14592 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48ba2211021f4aaba9a6ae3342963827 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:31:44.164865 14592 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48ba2211021f4aaba9a6ae3342963827 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:31:44.165079 14592 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48ba2211021f4aaba9a6ae3342963827 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:31:44.167271 14592 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48ba2211021f4aaba9a6ae3342963827 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48ba2211021f4aaba9a6ae3342963827" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 36267 } }
I20260926 16:31:44.167703 14592 leader_election.cc:302] T 00000000000000000000000000000000 P 48ba2211021f4aaba9a6ae3342963827 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48ba2211021f4aaba9a6ae3342963827; no voters:
I20260926 16:31:44.168208 14592 leader_election.cc:288] T 00000000000000000000000000000000 P 48ba2211021f4aaba9a6ae3342963827 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:31:44.168354 14597 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48ba2211021f4aaba9a6ae3342963827 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:31:44.169018 14597 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48ba2211021f4aaba9a6ae3342963827 [term 1 LEADER]: Becoming Leader. State: Replica: 48ba2211021f4aaba9a6ae3342963827, State: Running, Role: LEADER
I20260926 16:31:44.169590 14597 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48ba2211021f4aaba9a6ae3342963827 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48ba2211021f4aaba9a6ae3342963827" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 36267 } }
I20260926 16:31:44.170364 14592 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48ba2211021f4aaba9a6ae3342963827 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:31:44.172500 14599 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48ba2211021f4aaba9a6ae3342963827 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48ba2211021f4aaba9a6ae3342963827. Latest consensus state: current_term: 1 leader_uuid: "48ba2211021f4aaba9a6ae3342963827" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48ba2211021f4aaba9a6ae3342963827" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 36267 } } }
I20260926 16:31:44.172967 14599 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48ba2211021f4aaba9a6ae3342963827 [sys.catalog]: This master's current role is: LEADER
I20260926 16:31:44.173384 14598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48ba2211021f4aaba9a6ae3342963827 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "48ba2211021f4aaba9a6ae3342963827" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48ba2211021f4aaba9a6ae3342963827" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 36267 } } }
I20260926 16:31:44.173835 14598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48ba2211021f4aaba9a6ae3342963827 [sys.catalog]: This master's current role is: LEADER
I20260926 16:31:44.175635 14603 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:31:44.180733 14603 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:31:44.190260 14603 catalog_manager.cc:1347] Generated new cluster ID: 1cdce165e8a142a886cb4a7d127a07f3
I20260926 16:31:44.190400 14603 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:31:44.200989 14603 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:31:44.202445 14603 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:31:44.214903 14603 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 48ba2211021f4aaba9a6ae3342963827: Generated new TSK 0
I20260926 16:31:44.216152 14603 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.33.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab.
Entry for principal kudu/127.6.33.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab.
I20260926 16:31:44.278882 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.1:0
--local_ip_for_outbound_sockets=127.6.33.1
--webserver_interface=127.6.33.1
--webserver_port=0
--tserver_master_addrs=127.6.33.62:36267
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab
--principal=kudu/127.6.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:45713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:31:44.612149 14618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:31:44.612582 14618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:31:44.612805 14618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:31:44.620379 14618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:31:44.620551 14618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:31:44.620774 14618 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.1
I20260926 16:31:44.630671 14618 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:45713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab
--principal=kudu/127.6.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.33.1
--webserver_port=0
--tserver_master_addrs=127.6.33.62:36267
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.1
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:31:44.632009 14618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:31:44.633386 14618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:31:44.646409 14623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:44.646340 14624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:44.646342 14626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:31:44.647500 14618 server_base.cc:1034] running on GCE node
I20260926 16:31:44.648111 14618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:31:44.649154 14618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:31:44.650406 14618 hybrid_clock.cc:648] HybridClock initialized: now 1790440304650338 us; error 61 us; skew 500 ppm
I20260926 16:31:44.650743 14618 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:31:44 dist-test-slave-khd3 krb5kdc[14512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440304, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:31:44.654024 14618 init.cc:377] Logged in from keytab as kudu/127.6.33.1@KRBTEST.COM (short username kudu)
I20260926 16:31:44.655589 14618 webserver.cc:460] Webserver started at http://127.6.33.1:38307/ using document root <none> and password file <none>
I20260926 16:31:44.656299 14618 fs_manager.cc:359] Metadata directory not provided
I20260926 16:31:44.656445 14618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:31:44.656819 14618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:31:44.659994 14618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-0/data/instance:
uuid: "47eec1f7da2542b3b0dcb930294d3127"
format_stamp: "Formatted at 2026-09-26 16:31:44 on dist-test-slave-khd3"
server_key: "c3a77671afc086c898177e63fa75bf1f"
server_key_iv: "6bd2254bed123a501ff282e70dafb359"
server_key_version: "encryptionkey@0"
I20260926 16:31:44.660871 14618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-0/wal/instance:
uuid: "47eec1f7da2542b3b0dcb930294d3127"
format_stamp: "Formatted at 2026-09-26 16:31:44 on dist-test-slave-khd3"
server_key: "c3a77671afc086c898177e63fa75bf1f"
server_key_iv: "6bd2254bed123a501ff282e70dafb359"
server_key_version: "encryptionkey@0"
I20260926 16:31:44.666152 14618 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 16:31:44.669831 14633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:44.671451 14618 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:31:44.671638 14618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-0/wal
uuid: "47eec1f7da2542b3b0dcb930294d3127"
format_stamp: "Formatted at 2026-09-26 16:31:44 on dist-test-slave-khd3"
server_key: "c3a77671afc086c898177e63fa75bf1f"
server_key_iv: "6bd2254bed123a501ff282e70dafb359"
server_key_version: "encryptionkey@0"
I20260926 16:31:44.671842 14618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:31:44.695597 14618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:31:44.700031 14618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:31:44.700398 14618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:31:44.701494 14618 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:31:44.702916 14618 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:31:44.703023 14618 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:44.703174 14618 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:31:44.703248 14618 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:44.736989 14618 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.1:40909
I20260926 16:31:44.737074 14745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.1:40909 every 8 connection(s)
I20260926 16:31:44.738375 14618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
I20260926 16:31:44.743863 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 14618
I20260926 16:31:44.744084 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-0/wal/instance
I20260926 16:31:44.744684 6276 external_mini_cluster.cc:1428] Setting key e98d5c5b85eaace2b23d5449d05f9535
Sep 26 16:31:44 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440304, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.1@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:31:44.759485 14748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:44.741155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:33579 (local address 127.6.33.62:36267)
0926 16:31:44.741606 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:44.741611 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:44.742676 (+ 1065us) server_negotiation.cc:407] Connection header received
0926 16:31:44.744655 (+ 1979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:44.744662 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:44.744827 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:44.744965 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:44.746738 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:44.747467 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:44.749128 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:44.749537 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:44.749677 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:44.753823 (+ 4146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:44.753830 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:44.753836 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:44.753875 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:44.756307 (+ 2432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:44.757173 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:44.757179 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:44.757182 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:44.757304 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:44.757742 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:44.757747 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:44.757750 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:44.758014 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:44.758166 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:44.758877 (+ 711us) server_negotiation.cc:299] Negotiation successful
0926 16:31:44.759101 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":240,"threads_started":1}
I20260926 16:31:44.760448 14749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:44.741798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36267 (local address 127.6.33.1:33579)
0926 16:31:44.742515 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:44.742554 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:44.744378 (+ 1824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:44.745241 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:44.745253 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:44.745871 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:44.746554 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:44.746587 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:44.747650 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:44.747661 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:44.748977 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:44.748990 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:44.749772 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:44.749781 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:44.750039 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:44.750770 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:44.750802 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:44.753590 (+ 2788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:31:44.756503 (+ 2913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:44.756512 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:44.756530 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:44.757048 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:44.757422 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:44.757428 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:44.757431 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:44.757622 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:44.758141 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:31:44.758149 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:31:44.758508 (+ 359us) client_negotiation.cc:769] Sending connection context
0926 16:31:44.758770 (+ 262us) client_negotiation.cc:240] Negotiation successful
0926 16:31:44.759319 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":267,"threads_started":1}
I20260926 16:31:44.762073 14746 heartbeater.cc:347] Connected to a master server at 127.6.33.62:36267
I20260926 16:31:44.762447 14746 heartbeater.cc:464] Registering TS with master...
I20260926 16:31:44.763384 14746 heartbeater.cc:511] Master 127.6.33.62:36267 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.33.2@KRBTEST.COM; defaulting to no policy
I20260926 16:31:44.766021 14557 ts_manager.cc:194] Registered new tserver with Master: 47eec1f7da2542b3b0dcb930294d3127 (127.6.33.1:40909)
I20260926 16:31:44.767532 14557 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.33.1@KRBTEST.COM'} at 127.6.33.1:33579
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab.
Entry for principal kudu/127.6.33.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab.
I20260926 16:31:44.796468 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.2:0
--local_ip_for_outbound_sockets=127.6.33.2
--webserver_interface=127.6.33.2
--webserver_port=0
--tserver_master_addrs=127.6.33.62:36267
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab
--principal=kudu/127.6.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:45713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:31:45.127949 14752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:31:45.128221 14752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:31:45.128396 14752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:31:45.135879 14752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:31:45.135998 14752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:31:45.136247 14752 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.2
I20260926 16:31:45.146377 14752 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:45713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab
--principal=kudu/127.6.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.33.2
--webserver_port=0
--tserver_master_addrs=127.6.33.62:36267
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.2
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:31:45.147799 14752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:31:45.149197 14752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:31:45.163097 14760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:45.163128 14757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:45.163097 14758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:31:45.164000 14752 server_base.cc:1034] running on GCE node
I20260926 16:31:45.164675 14752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:31:45.165777 14752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:31:45.167037 14752 hybrid_clock.cc:648] HybridClock initialized: now 1790440305166999 us; error 38 us; skew 500 ppm
I20260926 16:31:45.167403 14752 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:31:45 dist-test-slave-khd3 krb5kdc[14512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440305, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:31:45.171020 14752 init.cc:377] Logged in from keytab as kudu/127.6.33.2@KRBTEST.COM (short username kudu)
I20260926 16:31:45.172616 14752 webserver.cc:460] Webserver started at http://127.6.33.2:46357/ using document root <none> and password file <none>
I20260926 16:31:45.173362 14752 fs_manager.cc:359] Metadata directory not provided
I20260926 16:31:45.173476 14752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:31:45.173736 14752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:31:45.177258 14752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-1/data/instance:
uuid: "19c33c59e5be4452833f15a6e38014b4"
format_stamp: "Formatted at 2026-09-26 16:31:45 on dist-test-slave-khd3"
server_key: "d9e0de3ee0557c4747e0d4d176b61df0"
server_key_iv: "a4e535ae6f9c2e9a661313bb6f9fe4c8"
server_key_version: "encryptionkey@0"
I20260926 16:31:45.178089 14752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-1/wal/instance:
uuid: "19c33c59e5be4452833f15a6e38014b4"
format_stamp: "Formatted at 2026-09-26 16:31:45 on dist-test-slave-khd3"
server_key: "d9e0de3ee0557c4747e0d4d176b61df0"
server_key_iv: "a4e535ae6f9c2e9a661313bb6f9fe4c8"
server_key_version: "encryptionkey@0"
I20260926 16:31:45.184022 14752 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260926 16:31:45.188043 14767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:45.189647 14752 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:31:45.189832 14752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-1/wal
uuid: "19c33c59e5be4452833f15a6e38014b4"
format_stamp: "Formatted at 2026-09-26 16:31:45 on dist-test-slave-khd3"
server_key: "d9e0de3ee0557c4747e0d4d176b61df0"
server_key_iv: "a4e535ae6f9c2e9a661313bb6f9fe4c8"
server_key_version: "encryptionkey@0"
I20260926 16:31:45.190048 14752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:31:45.218323 14752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:31:45.223120 14752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:31:45.223445 14752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:31:45.224509 14752 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:31:45.225999 14752 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:31:45.226071 14752 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:45.226199 14752 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:31:45.226249 14752 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:45.260160 14752 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.2:43561
I20260926 16:31:45.260196 14879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.2:43561 every 8 connection(s)
I20260926 16:31:45.261575 14752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
I20260926 16:31:45.269102 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 14752
I20260926 16:31:45.269294 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-1/wal/instance
I20260926 16:31:45.269757 6276 external_mini_cluster.cc:1428] Setting key f3caf414ca7f566d6dcafefb5c9c37da
Sep 26 16:31:45 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440305, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.2@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:31:45.283502 14882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:45.264423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:58263 (local address 127.6.33.62:36267)
0926 16:31:45.265173 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:45.265183 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:45.265774 (+ 591us) server_negotiation.cc:407] Connection header received
0926 16:31:45.267693 (+ 1919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:45.267699 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:45.267804 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:45.267923 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:45.269661 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:45.270519 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:45.272283 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:45.272703 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:45.272885 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:45.276823 (+ 3938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:45.276828 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:45.276832 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:45.276863 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:45.279684 (+ 2821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:45.280745 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:45.280820 (+ 75us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:45.280827 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:45.280992 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:45.281529 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:45.281534 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:45.281536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:45.281937 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:45.282061 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:45.283036 (+ 975us) server_negotiation.cc:299] Negotiation successful
0926 16:31:45.283237 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":495,"threads_started":1}
I20260926 16:31:45.285203 14883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:45.264807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36267 (local address 127.6.33.2:58263)
0926 16:31:45.265597 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:45.265646 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:45.267404 (+ 1758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:45.268167 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:45.268176 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:45.268721 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:45.269496 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:45.269515 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:45.270686 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:45.270693 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:45.272133 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:45.272145 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:45.272973 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:45.272980 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:45.273206 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:45.273931 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:45.273954 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:45.276526 (+ 2572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:31:45.279854 (+ 3328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:45.279866 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:45.279890 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:45.280564 (+ 674us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:45.281118 (+ 554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:45.281125 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:45.281132 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:45.281387 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:45.282090 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:31:45.282102 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:31:45.282624 (+ 522us) client_negotiation.cc:769] Sending connection context
0926 16:31:45.282951 (+ 327us) client_negotiation.cc:240] Negotiation successful
0926 16:31:45.283550 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":317,"threads_started":1}
I20260926 16:31:45.286775 14880 heartbeater.cc:347] Connected to a master server at 127.6.33.62:36267
I20260926 16:31:45.287154 14880 heartbeater.cc:464] Registering TS with master...
I20260926 16:31:45.287928 14880 heartbeater.cc:511] Master 127.6.33.62:36267 requested a full tablet report, sending...
I20260926 16:31:45.289717 14557 ts_manager.cc:194] Registered new tserver with Master: 19c33c59e5be4452833f15a6e38014b4 (127.6.33.2:43561)
I20260926 16:31:45.290575 14557 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.33.2@KRBTEST.COM'} at 127.6.33.2:58263
WARNING: no policy specified for kudu/127.6.33.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab.
Entry for principal kudu/127.6.33.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab.
I20260926 16:31:45.325695 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.3:0
--local_ip_for_outbound_sockets=127.6.33.3
--webserver_interface=127.6.33.3
--webserver_port=0
--tserver_master_addrs=127.6.33.62:36267
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab
--principal=kudu/127.6.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:45713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:31:45.664018 14886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:31:45.664397 14886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:31:45.664566 14886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:31:45.672241 14886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:31:45.672408 14886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:31:45.672600 14886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.3
I20260926 16:31:45.683183 14886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:45713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab
--principal=kudu/127.6.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.33.3
--webserver_port=0
--tserver_master_addrs=127.6.33.62:36267
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.3
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:31:45.684629 14886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:31:45.686048 14886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:31:45.700354 14891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:45.700659 14894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:45.700662 14892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:31:45.702045 14886 server_base.cc:1034] running on GCE node
I20260926 16:31:45.702785 14886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:31:45.703855 14886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:31:45.705072 14886 hybrid_clock.cc:648] HybridClock initialized: now 1790440305705000 us; error 63 us; skew 500 ppm
I20260926 16:31:45.705432 14886 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:31:45 dist-test-slave-khd3 krb5kdc[14512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440305, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:31:45.708948 14886 init.cc:377] Logged in from keytab as kudu/127.6.33.3@KRBTEST.COM (short username kudu)
I20260926 16:31:45.710561 14886 webserver.cc:460] Webserver started at http://127.6.33.3:34119/ using document root <none> and password file <none>
I20260926 16:31:45.711283 14886 fs_manager.cc:359] Metadata directory not provided
I20260926 16:31:45.711397 14886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:31:45.711661 14886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:31:45.714810 14886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-2/data/instance:
uuid: "ada280b255574668903f499c646e15bf"
format_stamp: "Formatted at 2026-09-26 16:31:45 on dist-test-slave-khd3"
server_key: "258a13c9447feb0ac49d71e10d1a3ff6"
server_key_iv: "7a41a8de71b6e434de9cce032d94e53e"
server_key_version: "encryptionkey@0"
I20260926 16:31:45.715659 14886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-2/wal/instance:
uuid: "ada280b255574668903f499c646e15bf"
format_stamp: "Formatted at 2026-09-26 16:31:45 on dist-test-slave-khd3"
server_key: "258a13c9447feb0ac49d71e10d1a3ff6"
server_key_iv: "7a41a8de71b6e434de9cce032d94e53e"
server_key_version: "encryptionkey@0"
I20260926 16:31:45.721458 14886 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260926 16:31:45.725206 14901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:45.726790 14886 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 16:31:45.727005 14886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-2/wal
uuid: "ada280b255574668903f499c646e15bf"
format_stamp: "Formatted at 2026-09-26 16:31:45 on dist-test-slave-khd3"
server_key: "258a13c9447feb0ac49d71e10d1a3ff6"
server_key_iv: "7a41a8de71b6e434de9cce032d94e53e"
server_key_version: "encryptionkey@0"
I20260926 16:31:45.727239 14886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:31:45.742297 14886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:31:45.747597 14886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:31:45.747946 14886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:31:45.749111 14886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:31:45.750552 14886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:31:45.750660 14886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:45.750820 14886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:31:45.750900 14886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:45.770838 14746 heartbeater.cc:503] Master 127.6.33.62:36267 was elected leader, sending a full tablet report...
I20260926 16:31:45.786155 14886 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.3:42631
I20260926 16:31:45.786168 15013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.3:42631 every 8 connection(s)
I20260926 16:31:45.787575 14886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
I20260926 16:31:45.788050 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 14886
I20260926 16:31:45.788275 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-2/wal/instance
I20260926 16:31:45.788892 6276 external_mini_cluster.cc:1428] Setting key 0fa039e36e55c120eeb75bcb273015dc
Sep 26 16:31:45 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440305, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.3@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:31:45.809832 15016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:45.791003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.3:60957 (local address 127.6.33.62:36267)
0926 16:31:45.791591 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:45.791595 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:45.792908 (+ 1313us) server_negotiation.cc:407] Connection header received
0926 16:31:45.794942 (+ 2034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:45.794947 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:45.795055 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:45.795176 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:45.797070 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:45.797867 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:45.799588 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:45.800000 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:45.800129 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:45.804101 (+ 3972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:45.804106 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:45.804109 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:45.804142 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:45.806893 (+ 2751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:45.807674 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:45.807677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:45.807679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:45.807799 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:45.808190 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:45.808193 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:45.808195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:45.808457 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:45.808564 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:45.809387 (+ 823us) server_negotiation.cc:299] Negotiation successful
0926 16:31:45.809564 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":362,"threads_started":1}
I20260926 16:31:45.810781 15017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:45.791735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36267 (local address 127.6.33.3:60957)
0926 16:31:45.792699 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:45.792740 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:45.794649 (+ 1909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:45.795452 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:45.795461 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:45.796039 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:45.796713 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:45.796731 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:45.798030 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:45.798036 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:45.799406 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:45.799419 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:45.800241 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:45.800248 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:45.800493 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:45.801290 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:45.801313 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:45.803857 (+ 2544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:31:45.807067 (+ 3210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:45.807075 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:45.807091 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:45.807554 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:45.807897 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:31:45.807900 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:31:45.807903 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:31:45.808092 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:31:45.808569 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:31:45.808575 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:31:45.809025 (+ 450us) client_negotiation.cc:769] Sending connection context
0926 16:31:45.809260 (+ 235us) client_negotiation.cc:240] Negotiation successful
0926 16:31:45.809663 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"thread_start_us":385,"threads_started":1}
I20260926 16:31:45.812433 15014 heartbeater.cc:347] Connected to a master server at 127.6.33.62:36267
I20260926 16:31:45.812855 15014 heartbeater.cc:464] Registering TS with master...
I20260926 16:31:45.813628 15014 heartbeater.cc:511] Master 127.6.33.62:36267 requested a full tablet report, sending...
I20260926 16:31:45.815199 14557 ts_manager.cc:194] Registered new tserver with Master: ada280b255574668903f499c646e15bf (127.6.33.3:42631)
I20260926 16:31:45.816246 14557 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.33.3@KRBTEST.COM'} at 127.6.33.3:60957
I20260926 16:31:45.817905 6276 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.6.33.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.33.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.33.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/krb5kdc/oryx_127.6.33.1.keytab.
Entry for principal oryx/127.6.33.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/krb5kdc/oryx_127.6.33.1.keytab.
WARNING: no policy specified for oryx/127.6.33.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.33.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.33.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/krb5kdc/oryx_127.6.33.2.keytab.
Entry for principal oryx/127.6.33.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/krb5kdc/oryx_127.6.33.2.keytab.
WARNING: no policy specified for oryx/127.6.33.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.33.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.33.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/krb5kdc/oryx_127.6.33.3.keytab.
Entry for principal oryx/127.6.33.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/krb5kdc/oryx_127.6.33.3.keytab.
I20260926 16:31:45.959707 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 14618
Sep 26 16:31:45 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.1@KRBTEST.COM
I20260926 16:31:45.977032 15025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:45.960949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53560 (local address 127.6.33.1:40909)
0926 16:31:45.962164 (+ 1215us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:45.962170 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:45.962197 (+ 27us) server_negotiation.cc:407] Connection header received
0926 16:31:45.962255 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:45.962261 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:45.962479 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:45.962616 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:45.963976 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:45.965063 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:45.966307 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:45.966871 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:45.966963 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:45.971213 (+ 4250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:45.971225 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:45.971240 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:45.971275 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:45.974266 (+ 2991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:45.974929 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:45.974936 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:45.974938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:45.975055 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:45.975423 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:45.975426 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:45.975428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:45.975829 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:45.976057 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:45.976359 (+ 302us) server_negotiation.cc:299] Negotiation successful
0926 16:31:45.976616 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1060,"thread_start_us":897,"threads_started":1}
I20260926 16:31:45.978340 14741 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:31:46.064129 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 14618
I20260926 16:31:46.080619 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 14752
Sep 26 16:31:46 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.2@KRBTEST.COM
I20260926 16:31:46.098591 15027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:46.081597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50846 (local address 127.6.33.2:43561)
0926 16:31:46.082177 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:46.082184 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:46.082257 (+ 73us) server_negotiation.cc:407] Connection header received
0926 16:31:46.082333 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:46.082339 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:46.082558 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:46.082687 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:46.083972 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:46.084997 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:46.086334 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:46.086893 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:46.086996 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:46.092362 (+ 5366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:46.092376 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:46.092418 (+ 42us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:46.092465 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:46.095532 (+ 3067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:46.096196 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:46.096203 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:46.096205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:46.096322 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:46.096707 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:46.096710 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:46.096712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:46.097240 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:46.097615 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:46.097926 (+ 311us) server_negotiation.cc:299] Negotiation successful
0926 16:31:46.098225 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":330,"threads_started":1}
I20260926 16:31:46.099900 14875 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:31:46.184840 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 14752
I20260926 16:31:46.200938 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 14886
Sep 26 16:31:46 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.3@KRBTEST.COM
I20260926 16:31:46.217638 15029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:46.201917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40092 (local address 127.6.33.3:42631)
0926 16:31:46.202533 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:46.202540 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:46.202568 (+ 28us) server_negotiation.cc:407] Connection header received
0926 16:31:46.202625 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:46.202631 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:46.202844 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:46.203028 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:46.204346 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:46.205441 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:46.206777 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:46.207345 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:46.207483 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:46.211885 (+ 4402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:46.211897 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:46.211913 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:46.211947 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:46.214998 (+ 3051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:46.215719 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:46.215727 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:46.215729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:46.215885 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:46.216203 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:46.216206 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:46.216208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:46.216565 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:46.216962 (+ 397us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:46.217113 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 16:31:46.217327 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":316,"threads_started":1}
I20260926 16:31:46.219146 15009 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:31:46.308177 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 14886
I20260926 16:31:46.324018 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 14525
I20260926 16:31:46.325340 14587 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:31:46.429564 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 14525
I20260926 16:31:46.450302 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:36267
--webserver_interface=127.6.33.62
--webserver_port=32935
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:45713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.33.62:36267
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:31:46.780692 15032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:31:46.781020 15032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:31:46.781113 15032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:31:46.788348 15032 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:31:46.788475 15032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:31:46.788524 15032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:31:46.788575 15032 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:31:46.788619 15032 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:31:46.798568 15032 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:45713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.33.62:36267
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:36267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.33.62
--webserver_port=32935
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:31:46.799891 15032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:31:46.801186 15032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:31:46.812675 15037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:46.812685 15038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:46.812796 15040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:31:46.813771 15032 server_base.cc:1034] running on GCE node
I20260926 16:31:46.814564 15032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:31:46.816118 15032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:31:46.817348 15032 hybrid_clock.cc:648] HybridClock initialized: now 1790440306817301 us; error 51 us; skew 500 ppm
I20260926 16:31:46.817709 15032 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:31:46 dist-test-slave-khd3 krb5kdc[14512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440306, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:31:46.821111 15032 init.cc:377] Logged in from keytab as kudu/127.6.33.62@KRBTEST.COM (short username kudu)
I20260926 16:31:46.822590 15032 webserver.cc:460] Webserver started at http://127.6.33.62:32935/ using document root <none> and password file <none>
I20260926 16:31:46.823289 15032 fs_manager.cc:359] Metadata directory not provided
I20260926 16:31:46.823454 15032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:31:46.828964 15032 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 16:31:46.831779 15047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:46.833186 15032 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:31:46.833371 15032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/master-0/wal
uuid: "48ba2211021f4aaba9a6ae3342963827"
format_stamp: "Formatted at 2026-09-26 16:31:44 on dist-test-slave-khd3"
server_key: "7d520954569ea777ade954a884151c6c"
server_key_iv: "9316669e46b1e140a39cef513012a121"
server_key_version: "encryptionkey@0"
I20260926 16:31:46.834338 15032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:31:46.854840 15032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:31:46.859591 15032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:31:46.859869 15032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:31:46.878887 15032 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.62:36267
I20260926 16:31:46.878904 15098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.62:36267 every 8 connection(s)
I20260926 16:31:46.880414 15032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
I20260926 16:31:46.882532 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 15032
I20260926 16:31:46.883335 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.1:40909
--local_ip_for_outbound_sockets=127.6.33.1
--tserver_master_addrs=127.6.33.62:36267
--webserver_port=38307
--webserver_interface=127.6.33.1
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab
--principal=kudu/127.6.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:45713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.6.33.1
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/krb5kdc/oryx_127.6.33.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:31:46.887614 15099 sys_catalog.cc:263] Verifying existing consensus state
I20260926 16:31:46.891721 15099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48ba2211021f4aaba9a6ae3342963827: Bootstrap starting.
I20260926 16:31:46.920269 15099 log.cc:824] T 00000000000000000000000000000000 P 48ba2211021f4aaba9a6ae3342963827: Log is configured to *not* fsync() on all Append() calls
I20260926 16:31:46.931299 15099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48ba2211021f4aaba9a6ae3342963827: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 16:31:46.931949 15099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48ba2211021f4aaba9a6ae3342963827: Bootstrap complete.
I20260926 16:31:46.945750 15099 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48ba2211021f4aaba9a6ae3342963827 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48ba2211021f4aaba9a6ae3342963827" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 36267 } }
I20260926 16:31:46.946966 15099 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48ba2211021f4aaba9a6ae3342963827 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48ba2211021f4aaba9a6ae3342963827, State: Initialized, Role: FOLLOWER
I20260926 16:31:46.947703 15099 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48ba2211021f4aaba9a6ae3342963827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48ba2211021f4aaba9a6ae3342963827" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 36267 } }
I20260926 16:31:46.948036 15099 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48ba2211021f4aaba9a6ae3342963827 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:31:46.948187 15099 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48ba2211021f4aaba9a6ae3342963827 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:31:46.948415 15099 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48ba2211021f4aaba9a6ae3342963827 [term 1 FOLLOWER]: Advancing to term 2
I20260926 16:31:46.955029 15099 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48ba2211021f4aaba9a6ae3342963827 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48ba2211021f4aaba9a6ae3342963827" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 36267 } }
I20260926 16:31:46.955639 15099 leader_election.cc:302] T 00000000000000000000000000000000 P 48ba2211021f4aaba9a6ae3342963827 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48ba2211021f4aaba9a6ae3342963827; no voters:
I20260926 16:31:46.956295 15099 leader_election.cc:288] T 00000000000000000000000000000000 P 48ba2211021f4aaba9a6ae3342963827 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 16:31:46.956457 15103 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48ba2211021f4aaba9a6ae3342963827 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 16:31:46.957703 15103 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48ba2211021f4aaba9a6ae3342963827 [term 2 LEADER]: Becoming Leader. State: Replica: 48ba2211021f4aaba9a6ae3342963827, State: Running, Role: LEADER
I20260926 16:31:46.958282 15103 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48ba2211021f4aaba9a6ae3342963827 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48ba2211021f4aaba9a6ae3342963827" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 36267 } }
I20260926 16:31:46.958927 15099 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48ba2211021f4aaba9a6ae3342963827 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:31:46.960939 15105 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48ba2211021f4aaba9a6ae3342963827 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48ba2211021f4aaba9a6ae3342963827. Latest consensus state: current_term: 2 leader_uuid: "48ba2211021f4aaba9a6ae3342963827" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48ba2211021f4aaba9a6ae3342963827" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 36267 } } }
I20260926 16:31:46.961320 15105 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48ba2211021f4aaba9a6ae3342963827 [sys.catalog]: This master's current role is: LEADER
I20260926 16:31:46.960913 15104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48ba2211021f4aaba9a6ae3342963827 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "48ba2211021f4aaba9a6ae3342963827" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48ba2211021f4aaba9a6ae3342963827" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 36267 } } }
I20260926 16:31:46.961894 15104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48ba2211021f4aaba9a6ae3342963827 [sys.catalog]: This master's current role is: LEADER
I20260926 16:31:46.970701 15109 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:31:46.977128 15109 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:31:46.978617 15109 catalog_manager.cc:1259] Loaded cluster ID: 1cdce165e8a142a886cb4a7d127a07f3
I20260926 16:31:46.978788 15109 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:31:46.982789 15109 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:31:46.984326 15109 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 48ba2211021f4aaba9a6ae3342963827: Loaded TSK: 0
I20260926 16:31:46.986099 15109 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 16:31:47.247817 15101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:31:47.248147 15101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:31:47.248306 15101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:31:47.255650 15101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:31:47.255820 15101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:31:47.255998 15101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.1
I20260926 16:31:47.266296 15101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:45713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/krb5kdc/oryx_127.6.33.1.keytab
--principal=oryx/127.6.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.1:40909
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.33.1
--webserver_port=38307
--tserver_master_addrs=127.6.33.62:36267
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.1
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:31:47.267676 15101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:31:47.269049 15101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:31:47.284416 15125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:47.284286 15126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:47.284324 15128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:31:47.286335 15101 server_base.cc:1034] running on GCE node
I20260926 16:31:47.287127 15101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:31:47.288311 15101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:31:47.289563 15101 hybrid_clock.cc:648] HybridClock initialized: now 1790440307289484 us; error 70 us; skew 500 ppm
I20260926 16:31:47.289923 15101 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:31:47 dist-test-slave-khd3 krb5kdc[14512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440307, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:31:47.293216 15101 init.cc:377] Logged in from keytab as oryx/127.6.33.1@KRBTEST.COM (short username oryx)
I20260926 16:31:47.294806 15101 webserver.cc:460] Webserver started at http://127.6.33.1:38307/ using document root <none> and password file <none>
I20260926 16:31:47.295542 15101 fs_manager.cc:359] Metadata directory not provided
I20260926 16:31:47.295683 15101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:31:47.301729 15101 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 16:31:47.304837 15135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:47.306370 15101 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 16:31:47.306553 15101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-0/wal
uuid: "47eec1f7da2542b3b0dcb930294d3127"
format_stamp: "Formatted at 2026-09-26 16:31:44 on dist-test-slave-khd3"
server_key: "c3a77671afc086c898177e63fa75bf1f"
server_key_iv: "6bd2254bed123a501ff282e70dafb359"
server_key_version: "encryptionkey@0"
I20260926 16:31:47.307523 15101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:31:47.323333 15101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:31:47.327832 15101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:31:47.328107 15101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:31:47.329276 15101 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:31:47.330534 15101 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:31:47.330618 15101 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:47.330700 15101 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:31:47.330760 15101 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:47.362955 15101 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.1:40909
I20260926 16:31:47.363060 15247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.1:40909 every 8 connection(s)
I20260926 16:31:47.364470 15101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
I20260926 16:31:47.369446 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 15101
I20260926 16:31:47.370223 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.2:43561
--local_ip_for_outbound_sockets=127.6.33.2
--tserver_master_addrs=127.6.33.62:36267
--webserver_port=46357
--webserver_interface=127.6.33.2
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab
--principal=kudu/127.6.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:45713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.6.33.2
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/krb5kdc/oryx_127.6.33.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 16:31:47 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.1@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:31:47 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.1@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:31:47.385329 15251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:47.367701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36267 (local address 127.6.33.1:58191)
0926 16:31:47.368495 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:47.368531 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:47.371084 (+ 2553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:47.372726 (+ 1642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:47.372739 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:47.373643 (+ 904us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:47.374476 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:47.374507 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:47.376266 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:47.376275 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:47.377800 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:47.377813 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:47.379459 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:47.379464 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:47.379670 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:47.380430 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:47.380458 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:47.384253 (+ 3795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.33.62:36267: Server oryx/127.6.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":358,"threads_started":1}
W20260926 16:31:47.385975 15250 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:31:47.367572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:58191 (local address 127.6.33.62:36267)
0926 16:31:47.368341 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:47.368359 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:47.368651 (+ 292us) server_negotiation.cc:407] Connection header received
0926 16:31:47.371755 (+ 3104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:47.371771 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:47.372174 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:47.372491 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:47.374713 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:47.376052 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:47.377957 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:47.379205 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:47.379414 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:47.384744 (+ 5330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.33.1:58191: BlockingRecv error: recv got EOF from 127.6.33.1:58191 (error 108)
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":223,"threads_started":1}
W20260926 16:31:47.386188 15248 heartbeater.cc:644] Failed to heartbeat to 127.6.33.62:36267 (0 consecutive failures): Not authorized: Failed to ping master at 127.6.33.62:36267: Client connection negotiation failed: client connection to 127.6.33.62:36267: Server oryx/127.6.33.62@KRBTEST.COM not found in Kerberos database
Sep 26 16:31:47 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.1@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:31:47 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.1@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:31:47.395711 15251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:47.386822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36267 (local address 127.6.33.1:53757)
0926 16:31:47.387031 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:47.387055 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:47.387168 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:47.387726 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:47.387730 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:47.388101 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:47.388236 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:47.388244 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:47.389836 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:47.389841 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:47.390706 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:47.390716 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:47.391780 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:47.391784 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:47.391904 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:47.392513 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:47.392529 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:47.395518 (+ 2989us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.33.62:36267: Server oryx/127.6.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":61}
W20260926 16:31:47.396013 15250 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:31:47.386925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:53757 (local address 127.6.33.62:36267)
0926 16:31:47.387391 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:47.387400 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:47.387419 (+ 19us) server_negotiation.cc:407] Connection header received
0926 16:31:47.387495 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:47.387503 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:47.387583 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:47.387689 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:47.388389 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:47.389692 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:47.390847 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:47.391637 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:47.391788 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:47.395782 (+ 3994us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.33.1:53757: BlockingRecv error: recv got EOF from 127.6.33.1:53757 (error 108)
Metrics: {"server-negotiator.queue_time_us":337}
Sep 26 16:31:47 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.1@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:31:47 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.1@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:31:47.404934 15251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:47.396435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36267 (local address 127.6.33.1:57255)
0926 16:31:47.396643 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:47.396663 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:47.396791 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:47.397362 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:47.397365 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:47.397624 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:47.397771 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:47.397779 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:47.399180 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:47.399184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:47.400112 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:47.400123 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:47.401044 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:47.401048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:47.401171 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:47.401781 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:47.401798 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:47.404605 (+ 2807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.33.62:36267: Server oryx/127.6.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":78,"thread_start_us":533,"threads_started":1}
W20260926 16:31:47.405192 15250 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:31:47.396682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:57255 (local address 127.6.33.62:36267)
0926 16:31:47.396919 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:47.396927 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:47.396951 (+ 24us) server_negotiation.cc:407] Connection header received
0926 16:31:47.397017 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:47.397024 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:47.397107 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:47.397221 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:47.397913 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:47.399032 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:47.400240 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:47.400905 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:47.401022 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:47.404930 (+ 3908us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.33.1:57255: BlockingRecv error: recv got EOF from 127.6.33.1:57255 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
W20260926 16:31:47.405344 15248 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 16:31:47.711403 15252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:31:47.711752 15252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:31:47.711956 15252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:31:47.720098 15252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:31:47.720281 15252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:31:47.720461 15252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.2
I20260926 16:31:47.730669 15252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:45713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/krb5kdc/oryx_127.6.33.2.keytab
--principal=oryx/127.6.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.2:43561
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.33.2
--webserver_port=46357
--tserver_master_addrs=127.6.33.62:36267
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.2
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:31:47.732076 15252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:31:47.733475 15252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:31:47.747037 15258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:47.747004 15261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:47.747074 15259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:31:47.747854 15252 server_base.cc:1034] running on GCE node
I20260926 16:31:47.748453 15252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:31:47.749461 15252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:31:47.750662 15252 hybrid_clock.cc:648] HybridClock initialized: now 1790440307750622 us; error 29 us; skew 500 ppm
I20260926 16:31:47.751098 15252 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:31:47 dist-test-slave-khd3 krb5kdc[14512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440307, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:31:47.754225 15252 init.cc:377] Logged in from keytab as oryx/127.6.33.2@KRBTEST.COM (short username oryx)
I20260926 16:31:47.755707 15252 webserver.cc:460] Webserver started at http://127.6.33.2:46357/ using document root <none> and password file <none>
I20260926 16:31:47.756398 15252 fs_manager.cc:359] Metadata directory not provided
I20260926 16:31:47.756546 15252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:31:47.762176 15252 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260926 16:31:47.764853 15268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:47.766388 15252 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 16:31:47.766592 15252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-1/wal
uuid: "19c33c59e5be4452833f15a6e38014b4"
format_stamp: "Formatted at 2026-09-26 16:31:45 on dist-test-slave-khd3"
server_key: "d9e0de3ee0557c4747e0d4d176b61df0"
server_key_iv: "a4e535ae6f9c2e9a661313bb6f9fe4c8"
server_key_version: "encryptionkey@0"
I20260926 16:31:47.767680 15252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:31:47.788403 15252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:31:47.792840 15252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:31:47.793135 15252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:31:47.794134 15252 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:31:47.795576 15252 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:31:47.795699 15252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:47.795832 15252 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:31:47.795938 15252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:47.828550 15252 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.2:43561
I20260926 16:31:47.828604 15380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.2:43561 every 8 connection(s)
I20260926 16:31:47.830034 15252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
I20260926 16:31:47.836617 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 15252
I20260926 16:31:47.837337 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.3:42631
--local_ip_for_outbound_sockets=127.6.33.3
--tserver_master_addrs=127.6.33.62:36267
--webserver_port=34119
--webserver_interface=127.6.33.3
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab
--principal=kudu/127.6.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:45713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.6.33.3
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/krb5kdc/oryx_127.6.33.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 16:31:47 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.2@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:31:47 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.2@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 16:31:47.848202 15250 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:31:47.832600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:47021 (local address 127.6.33.62:36267)
0926 16:31:47.832827 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:47.832832 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:47.834034 (+ 1202us) server_negotiation.cc:407] Connection header received
0926 16:31:47.835773 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:47.835777 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:47.835843 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:47.835932 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:47.838182 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:47.839069 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:47.841755 (+ 2686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:47.842178 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:47.842282 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:47.847958 (+ 5676us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.33.2:47021: BlockingRecv error: recv got EOF from 127.6.33.2:47021 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 16:31:47.849069 15383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:47.833157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36267 (local address 127.6.33.2:47021)
0926 16:31:47.833865 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:47.833910 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:47.835513 (+ 1603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:47.836175 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:47.836187 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:47.837017 (+ 830us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:47.837974 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:47.838004 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:47.839290 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:47.839298 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:47.841581 (+ 2283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:47.841601 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:47.842425 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:47.842434 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:47.842795 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:47.843607 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:47.843642 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:47.847781 (+ 4139us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.33.62:36267: Server oryx/127.6.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":286,"threads_started":1}
W20260926 16:31:47.849902 15381 heartbeater.cc:644] Failed to heartbeat to 127.6.33.62:36267 (0 consecutive failures): Not authorized: Failed to ping master at 127.6.33.62:36267: Client connection negotiation failed: client connection to 127.6.33.62:36267: Server oryx/127.6.33.62@KRBTEST.COM not found in Kerberos database
Sep 26 16:31:47 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.2@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:31:47 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.2@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:31:47.859333 15383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:47.850657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36267 (local address 127.6.33.2:58783)
0926 16:31:47.850859 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:47.850875 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:47.850969 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:47.851299 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:47.851302 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:47.851608 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:47.851764 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:47.851773 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:47.853306 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:47.853311 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:47.854218 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:47.854229 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:47.855299 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:47.855303 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:47.855440 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:47.856055 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:47.856072 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:47.859036 (+ 2964us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.33.62:36267: Server oryx/127.6.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":74}
W20260926 16:31:47.859645 15250 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:31:47.850758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:58783 (local address 127.6.33.62:36267)
0926 16:31:47.850970 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:47.850978 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:47.851001 (+ 23us) server_negotiation.cc:407] Connection header received
0926 16:31:47.851061 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:47.851068 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:47.851160 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:47.851265 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:47.851917 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:47.853145 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:47.854378 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:47.855115 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:47.855280 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:47.859347 (+ 4067us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.33.2:58783: BlockingRecv error: recv got EOF from 127.6.33.2:58783 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 26 16:31:47 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.2@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:31:47 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.2@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:31:47.868288 15383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:47.860185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36267 (local address 127.6.33.2:53009)
0926 16:31:47.860450 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:47.860469 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:47.860606 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:47.860978 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:47.860982 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:47.861294 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:47.861433 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:47.861441 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:47.862394 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:47.862401 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:47.863295 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:47.863305 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:47.864039 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:47.864046 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:47.864228 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:47.864821 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:47.864843 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:47.867980 (+ 3137us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.33.62:36267: Server oryx/127.6.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 16:31:47.868553 15381 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 16:31:47.868829 15250 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:31:47.860307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:53009 (local address 127.6.33.62:36267)
0926 16:31:47.860498 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:47.860505 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:47.860543 (+ 38us) server_negotiation.cc:407] Connection header received
0926 16:31:47.860728 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:47.860733 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:47.860855 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:47.860956 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:47.861567 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:47.862258 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:47.863433 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:47.863883 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:47.863998 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:47.868178 (+ 4180us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.33.2:53009: BlockingRecv error: recv got EOF from 127.6.33.2:53009 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260926 16:31:48.171468 15384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:31:48.171854 15384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:31:48.172021 15384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:31:48.179309 15384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:31:48.179463 15384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:31:48.179641 15384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.3
I20260926 16:31:48.189538 15384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:45713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/krb5kdc/oryx_127.6.33.3.keytab
--principal=oryx/127.6.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.3:42631
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.33.3
--webserver_port=34119
--tserver_master_addrs=127.6.33.62:36267
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.3
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:31:48.190989 15384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:31:48.192370 15384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:31:48.205770 15390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:48.205720 15389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:31:48.205744 15392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:31:48.206714 15384 server_base.cc:1034] running on GCE node
I20260926 16:31:48.207293 15384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:31:48.208233 15384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:31:48.209427 15384 hybrid_clock.cc:648] HybridClock initialized: now 1790440308209387 us; error 31 us; skew 500 ppm
I20260926 16:31:48.209754 15384 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:31:48 dist-test-slave-khd3 krb5kdc[14512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440308, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:31:48.213122 15384 init.cc:377] Logged in from keytab as oryx/127.6.33.3@KRBTEST.COM (short username oryx)
I20260926 16:31:48.214514 15384 webserver.cc:460] Webserver started at http://127.6.33.3:34119/ using document root <none> and password file <none>
I20260926 16:31:48.215183 15384 fs_manager.cc:359] Metadata directory not provided
I20260926 16:31:48.215293 15384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:31:48.220623 15384 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260926 16:31:48.223652 15399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:48.225224 15384 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:31:48.225410 15384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-2/wal
uuid: "ada280b255574668903f499c646e15bf"
format_stamp: "Formatted at 2026-09-26 16:31:45 on dist-test-slave-khd3"
server_key: "258a13c9447feb0ac49d71e10d1a3ff6"
server_key_iv: "7a41a8de71b6e434de9cce032d94e53e"
server_key_version: "encryptionkey@0"
I20260926 16:31:48.226398 15384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:31:48.271231 15384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:31:48.275871 15384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:31:48.276170 15384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:31:48.277241 15384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:31:48.278699 15384 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:31:48.278798 15384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:48.278959 15384 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:31:48.279038 15384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:31:48.312073 15384 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.3:42631
I20260926 16:31:48.312186 15511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.3:42631 every 8 connection(s)
I20260926 16:31:48.313524 15384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
I20260926 16:31:48.323810 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 15384
Sep 26 16:31:48 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.3@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:31:48 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.3@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 16:31:48.329558 15250 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:31:48.316028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.3:38513 (local address 127.6.33.62:36267)
0926 16:31:48.316208 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:48.316213 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:48.317430 (+ 1217us) server_negotiation.cc:407] Connection header received
0926 16:31:48.319238 (+ 1808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:48.319243 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:48.319341 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:48.319448 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:48.321003 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:48.321700 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:48.323210 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:48.323869 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:48.324005 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:48.329372 (+ 5367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.33.3:38513: BlockingRecv error: recv got EOF from 127.6.33.3:38513 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 16:31:48.330483 15514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:48.316537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36267 (local address 127.6.33.3:38513)
0926 16:31:48.317265 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:48.317302 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:48.318969 (+ 1667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:48.319677 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:48.319685 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:48.320208 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:48.320752 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:48.320856 (+ 104us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:48.321848 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:48.321853 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:48.323063 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:48.323073 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:48.324098 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:48.324104 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:48.324302 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:48.325085 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:48.325120 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:48.329232 (+ 4112us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.33.62:36267: Server oryx/127.6.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":298,"threads_started":1}
W20260926 16:31:48.331236 15512 heartbeater.cc:644] Failed to heartbeat to 127.6.33.62:36267 (0 consecutive failures): Not authorized: Failed to ping master at 127.6.33.62:36267: Client connection negotiation failed: client connection to 127.6.33.62:36267: Server oryx/127.6.33.62@KRBTEST.COM not found in Kerberos database
Sep 26 16:31:48 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.3@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:31:48 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.3@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:31:48.340713 15514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:48.331841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36267 (local address 127.6.33.3:44615)
0926 16:31:48.331999 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:48.332016 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:48.332106 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:48.332457 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:48.332460 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:48.332841 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:48.332995 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:48.333004 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:48.334240 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:48.334248 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:48.335771 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:48.335785 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:48.336564 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:48.336570 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:48.336741 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:48.337264 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:48.337289 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:48.340418 (+ 3129us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.33.62:36267: Server oryx/127.6.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":57}
W20260926 16:31:48.340901 15250 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:31:48.331923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.3:44615 (local address 127.6.33.62:36267)
0926 16:31:48.332103 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:48.332110 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:48.332131 (+ 21us) server_negotiation.cc:407] Connection header received
0926 16:31:48.332184 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:48.332190 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:48.332266 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:48.332393 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:48.333132 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:48.334083 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:48.335967 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:48.336396 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:48.336508 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:48.340601 (+ 4093us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.33.3:44615: BlockingRecv error: recv got EOF from 127.6.33.3:44615 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 16:31:48.341976 15516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:48.324895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36278 (local address 127.6.33.62:36267)
0926 16:31:48.325603 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:48.325609 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:48.325630 (+ 21us) server_negotiation.cc:407] Connection header received
0926 16:31:48.325675 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:48.325680 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:48.325747 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:48.325857 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:48.327302 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:48.328129 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:48.329902 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:48.330365 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:48.330456 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:48.333736 (+ 3280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:31:48.333745 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:31:48.333763 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:31:48.333806 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:31:48.337885 (+ 4079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:48.338789 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:48.338804 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:48.338812 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:48.339079 (+ 267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:31:48.339491 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:31:48.339496 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:31:48.339499 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:31:48.340043 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:31:48.340356 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0926 16:31:48.340864 (+ 508us) server_negotiation.cc:299] Negotiation successful
0926 16:31:48.341265 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":476,"threads_started":1}
Sep 26 16:31:48 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.3@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:31:48 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.3@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:31:48.350114 15514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:48.341652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36267 (local address 127.6.33.3:53765)
0926 16:31:48.341799 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:48.341817 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:48.341908 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:48.342357 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:48.342362 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:48.342667 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:48.342811 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:48.342819 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:48.344518 (+ 1699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:48.344524 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:48.345604 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:48.345616 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:48.346384 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:48.346388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:48.346505 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:48.347101 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:48.347117 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:48.349910 (+ 2793us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.33.62:36267: Server oryx/127.6.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 16:31:48.350405 15512 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 16:31:48.350477 15250 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:31:48.341840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.3:53765 (local address 127.6.33.62:36267)
0926 16:31:48.342095 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:48.342102 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:48.342122 (+ 20us) server_negotiation.cc:407] Connection header received
0926 16:31:48.342165 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:48.342170 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:48.342230 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:48.342344 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:48.342949 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:48.344113 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:48.345779 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:48.346228 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:48.346328 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:48.350088 (+ 3760us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.33.3:53765: BlockingRecv error: recv got EOF from 127.6.33.3:53765 (error 108)
Metrics: {"server-negotiator.queue_time_us":134}
Sep 26 16:31:48 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.1@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:31:48 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.1@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:31:48.414772 15517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:48.405983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36267 (local address 127.6.33.1:39603)
0926 16:31:48.406503 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:48.406519 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:48.406700 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:48.407125 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:48.407129 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:48.407701 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:48.407897 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:48.407905 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:48.408918 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:48.408922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:48.409907 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:48.409918 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:48.410673 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:48.410678 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:48.410797 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:48.411530 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:48.411548 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:48.414496 (+ 2948us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.33.62:36267: Server oryx/127.6.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":263,"threads_started":1}
W20260926 16:31:48.415077 15250 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:31:48.406058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:39603 (local address 127.6.33.62:36267)
0926 16:31:48.406282 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:48.406288 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:48.406605 (+ 317us) server_negotiation.cc:407] Connection header received
0926 16:31:48.406862 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:48.406867 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:48.406955 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:48.407066 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:48.408037 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:48.408784 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:48.410063 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:48.410521 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:48.410618 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:48.414723 (+ 4105us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.33.1:39603: BlockingRecv error: recv got EOF from 127.6.33.1:39603 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 26 16:31:48 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.2@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:31:48 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.2@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:31:48.878361 15518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:48.869334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36267 (local address 127.6.33.2:46455)
0926 16:31:48.869914 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:48.869929 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:48.870090 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:48.870490 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:48.870493 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:48.871049 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:48.871362 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:48.871385 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:48.872685 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:48.872689 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:48.873770 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:48.873782 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:48.874473 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:48.874477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:48.874596 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:48.875244 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:48.875261 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:48.878068 (+ 2807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.33.62:36267: Server oryx/127.6.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":306,"threads_started":1}
W20260926 16:31:48.878571 15250 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:31:48.869340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:46455 (local address 127.6.33.62:36267)
0926 16:31:48.869536 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:48.869540 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:48.870018 (+ 478us) server_negotiation.cc:407] Connection header received
0926 16:31:48.870260 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:48.870264 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:48.870328 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:48.870455 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:48.871529 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:48.872536 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:48.873908 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:48.874297 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:48.874401 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:48.878308 (+ 3907us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.33.2:46455: BlockingRecv error: recv got EOF from 127.6.33.2:46455 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 26 16:31:49 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.3@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:31:49 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.3@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:31:49.367401 15519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:49.351063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36267 (local address 127.6.33.3:52163)
0926 16:31:49.351574 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:49.351590 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:49.351748 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:49.352105 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:49.352110 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:49.352570 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:49.352804 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:49.352823 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:49.353861 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:49.353868 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:49.354800 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:49.354811 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:49.355579 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:49.355587 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:49.355744 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:49.363812 (+ 8068us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:49.363832 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:49.367162 (+ 3330us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.33.62:36267: Server oryx/127.6.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":279,"threads_started":1}
W20260926 16:31:49.367579 15250 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:31:49.351133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.3:52163 (local address 127.6.33.62:36267)
0926 16:31:49.351391 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:49.351400 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:49.351678 (+ 278us) server_negotiation.cc:407] Connection header received
0926 16:31:49.351890 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:49.351896 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:49.351967 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:49.352070 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:49.352955 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:49.353700 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:49.354960 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:49.355364 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:49.355519 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:49.367333 (+ 11814us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.33.3:52163: BlockingRecv error: recv got EOF from 127.6.33.3:52163 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 26 16:31:49 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.1@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:31:49 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.1@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:31:49.424129 15520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:49.415790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36267 (local address 127.6.33.1:59423)
0926 16:31:49.416392 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:49.416407 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:49.416562 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:49.417034 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:49.417039 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:49.417476 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:49.417657 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:49.417666 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:49.418717 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:49.418721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:49.419649 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:49.419660 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:49.420305 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:49.420309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:49.420445 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:49.421108 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:49.421125 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:49.423762 (+ 2637us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.33.62:36267: Server oryx/127.6.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":275,"threads_started":1}
W20260926 16:31:49.424198 15250 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:31:49.415875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:59423 (local address 127.6.33.62:36267)
0926 16:31:49.416138 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:49.416145 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:49.416489 (+ 344us) server_negotiation.cc:407] Connection header received
0926 16:31:49.416705 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:49.416709 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:49.416827 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:49.416940 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:49.417785 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:49.418576 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:49.419782 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:49.420162 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:49.420270 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:49.423954 (+ 3684us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.33.1:59423: BlockingRecv error: recv got EOF from 127.6.33.1:59423 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 26 16:31:49 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.2@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:31:49 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.2@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:31:49.888700 15521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:49.879379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36267 (local address 127.6.33.2:33299)
0926 16:31:49.879970 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:49.879989 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:49.880113 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:49.880423 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:49.880429 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:49.880949 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:49.881125 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:49.881136 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:49.882266 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:49.882271 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:49.883763 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:49.883779 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:49.884530 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:49.884536 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:49.884700 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:49.885354 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:49.885377 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:49.888446 (+ 3069us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.33.62:36267: Server oryx/127.6.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":348,"threads_started":1}
W20260926 16:31:49.888882 15250 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:31:49.879432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:33299 (local address 127.6.33.62:36267)
0926 16:31:49.879634 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:49.879643 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:49.880073 (+ 430us) server_negotiation.cc:407] Connection header received
0926 16:31:49.880216 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:49.880220 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:49.880280 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:49.880386 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:49.881268 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:49.882125 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:49.883923 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:49.884380 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:49.884478 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:49.888639 (+ 4161us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.33.2:33299: BlockingRecv error: recv got EOF from 127.6.33.2:33299 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 26 16:31:50 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.3@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:31:50 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.3@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:31:50.376963 15522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:50.368337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36267 (local address 127.6.33.3:34615)
0926 16:31:50.368898 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:50.368914 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:50.369044 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:50.369492 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:50.369496 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:50.369893 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:50.370030 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:50.370038 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:50.371105 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:50.371109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:50.372019 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:50.372029 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:50.372893 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:50.372898 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:50.373003 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:50.373585 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:50.373602 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:50.376587 (+ 2985us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.33.62:36267: Server oryx/127.6.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":286,"threads_started":1}
W20260926 16:31:50.377058 15250 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:31:50.368516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.3:34615 (local address 127.6.33.62:36267)
0926 16:31:50.368685 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:50.368690 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:50.368973 (+ 283us) server_negotiation.cc:407] Connection header received
0926 16:31:50.369205 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:50.369210 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:50.369292 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:50.369395 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:50.370167 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:50.370985 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:50.372245 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:50.372694 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:50.372855 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:50.376802 (+ 3947us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.33.3:34615: BlockingRecv error: recv got EOF from 127.6.33.3:34615 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 26 16:31:50 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.1@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:31:50 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.1@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:31:50.433629 15523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:50.425196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36267 (local address 127.6.33.1:58123)
0926 16:31:50.425743 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:50.425763 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:50.425928 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:50.426307 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:50.426312 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:50.426639 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:50.426819 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:50.426826 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:50.427787 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:50.427791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:50.428830 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:50.428842 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:50.429542 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:50.429546 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:50.429663 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:50.430344 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:50.430379 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:50.433258 (+ 2879us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.33.62:36267: Server oryx/127.6.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":262,"threads_started":1}
W20260926 16:31:50.433712 15250 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:31:50.425233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:58123 (local address 127.6.33.62:36267)
0926 16:31:50.425473 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:50.425482 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:50.425874 (+ 392us) server_negotiation.cc:407] Connection header received
0926 16:31:50.426050 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:50.426055 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:50.426155 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:50.426263 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:50.426948 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:50.427668 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:50.428960 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:50.429424 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:50.429526 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:50.433476 (+ 3950us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.33.1:58123: BlockingRecv error: recv got EOF from 127.6.33.1:58123 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 26 16:31:50 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.2@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:31:50 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.2@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:31:50.898666 15524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:50.889733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36267 (local address 127.6.33.2:55389)
0926 16:31:50.890342 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:50.890357 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:50.890537 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:50.890906 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:50.890910 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:50.891283 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:50.891446 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:50.891454 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:50.892580 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:50.892586 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:50.893611 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:50.893623 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:50.894334 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:50.894339 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:50.894517 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:50.895190 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:50.895207 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:50.898414 (+ 3207us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.33.62:36267: Server oryx/127.6.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":334,"threads_started":1}
W20260926 16:31:50.898973 15250 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:31:50.889930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:55389 (local address 127.6.33.62:36267)
0926 16:31:50.890208 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:50.890215 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:50.890450 (+ 235us) server_negotiation.cc:407] Connection header received
0926 16:31:50.890690 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:50.890695 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:50.890765 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:50.890879 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:50.891601 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:50.892439 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:50.893767 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:50.894183 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:50.894318 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:50.898703 (+ 4385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.33.2:55389: BlockingRecv error: recv got EOF from 127.6.33.2:55389 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Sep 26 16:31:51 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.3@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:31:51 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.3@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:31:51.386411 15525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:51.378046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36267 (local address 127.6.33.3:57979)
0926 16:31:51.378836 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:51.378857 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:51.378997 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:51.379487 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:51.379492 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:51.379968 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:51.380139 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:51.380158 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:51.381151 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:51.381155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:51.382084 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:51.382096 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:51.382735 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:51.382739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:51.382859 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:51.383452 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:51.383469 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:51.386146 (+ 2677us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.33.62:36267: Server oryx/127.6.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":670,"thread_start_us":507,"threads_started":1}
W20260926 16:31:51.386715 15250 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:31:51.378299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.3:57979 (local address 127.6.33.62:36267)
0926 16:31:51.378541 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:51.378550 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:51.378971 (+ 421us) server_negotiation.cc:407] Connection header received
0926 16:31:51.379163 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:51.379171 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:51.379316 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:51.379402 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:51.380279 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:51.381018 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:51.382226 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:51.382615 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:51.382716 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:51.386393 (+ 3677us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.33.3:57979: BlockingRecv error: recv got EOF from 127.6.33.3:57979 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 26 16:31:51 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.1@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:31:51 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.1@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:31:51.443063 15526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:51.434622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36267 (local address 127.6.33.1:53013)
0926 16:31:51.435165 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:51.435181 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:51.435291 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:51.435571 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:51.435574 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:51.435935 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:51.436075 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:51.436083 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:51.437099 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:51.437104 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:51.438282 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:51.438295 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:51.439173 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:51.439179 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:51.439364 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:51.440064 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:51.440088 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:51.442845 (+ 2757us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.33.62:36267: Server oryx/127.6.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":309,"threads_started":1}
W20260926 16:31:51.443337 15250 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:31:51.434719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:53013 (local address 127.6.33.62:36267)
0926 16:31:51.434914 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:51.434919 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:51.435250 (+ 331us) server_negotiation.cc:407] Connection header received
0926 16:31:51.435385 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:51.435389 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:51.435447 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:51.435529 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:51.436196 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:51.436962 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:51.438429 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:51.439043 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:51.439164 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:51.443043 (+ 3879us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.33.1:53013: BlockingRecv error: recv got EOF from 127.6.33.1:53013 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 26 16:31:51 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.2@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:31:51 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.2@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:31:51.908206 15527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:51.899766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36267 (local address 127.6.33.2:40701)
0926 16:31:51.900345 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:51.900361 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:51.900489 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:51.900877 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:51.900882 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:51.901220 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:51.901375 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:51.901384 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:51.902334 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:51.902338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:51.903509 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:51.903520 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:51.904402 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:51.904406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:51.904541 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:51.905233 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:51.905250 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:51.907981 (+ 2731us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.33.62:36267: Server oryx/127.6.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":334,"threads_started":1}
W20260926 16:31:51.908384 15250 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:31:51.899848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:40701 (local address 127.6.33.62:36267)
0926 16:31:51.900024 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:51.900033 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:51.900444 (+ 411us) server_negotiation.cc:407] Connection header received
0926 16:31:51.900599 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:51.900602 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:51.900681 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:51.900851 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:51.901505 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:51.902212 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:51.903676 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:51.904242 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:51.904387 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:51.908178 (+ 3791us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.33.2:40701: BlockingRecv error: recv got EOF from 127.6.33.2:40701 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 26 16:31:52 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.3@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:31:52 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.3@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:31:52.397032 15528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:52.387475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36267 (local address 127.6.33.3:42577)
0926 16:31:52.388035 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:52.388053 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:52.388269 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:52.388658 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:52.388664 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:52.389110 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:52.389278 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:52.389287 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:52.390782 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:52.390789 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:52.391961 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:52.391973 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:52.392750 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:52.392783 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:52.392980 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:52.393584 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:52.393603 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:52.396626 (+ 3023us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.33.62:36267: Server oryx/127.6.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":334,"threads_started":1}
W20260926 16:31:52.397166 15250 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:31:52.387596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.3:42577 (local address 127.6.33.62:36267)
0926 16:31:52.387850 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:52.387860 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:52.388146 (+ 286us) server_negotiation.cc:407] Connection header received
0926 16:31:52.388431 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:52.388439 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:52.388512 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:52.388626 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:52.389449 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:52.390633 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:52.392110 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:52.392572 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:52.392738 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:52.396907 (+ 4169us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.33.3:42577: BlockingRecv error: recv got EOF from 127.6.33.3:42577 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 26 16:31:52 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.1@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:31:52 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.1@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:31:52.452967 15529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:52.444044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36267 (local address 127.6.33.1:40407)
0926 16:31:52.444598 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:52.444614 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:52.444750 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:52.445115 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:52.445119 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:52.445515 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:52.445677 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:52.445685 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:52.446938 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:52.446942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:52.447843 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:52.447853 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:52.448492 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:52.448496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:52.448608 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:52.449416 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:52.449433 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:52.452658 (+ 3225us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.33.62:36267: Server oryx/127.6.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":292,"threads_started":1}
W20260926 16:31:52.453083 15250 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:31:52.444166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:40407 (local address 127.6.33.62:36267)
0926 16:31:52.444343 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:52.444348 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:52.444699 (+ 351us) server_negotiation.cc:407] Connection header received
0926 16:31:52.444884 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:52.444887 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:52.444949 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:52.445035 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:52.445821 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:52.446782 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:52.447979 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:52.448371 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:52.448462 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:52.452906 (+ 4444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.33.1:40407: BlockingRecv error: recv got EOF from 127.6.33.1:40407 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 26 16:31:52 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.2@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:31:52 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.2@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:31:52.917814 15530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:52.909252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36267 (local address 127.6.33.2:45773)
0926 16:31:52.909823 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:52.909857 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:52.909972 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:52.910272 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:52.910275 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:52.910656 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:52.910842 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:52.910852 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:52.911941 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:52.911945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:52.913168 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:52.913204 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:52.914017 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:52.914023 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:52.914194 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:52.914843 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:52.914861 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:52.917579 (+ 2718us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.33.62:36267: Server oryx/127.6.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":307,"threads_started":1}
W20260926 16:31:52.918026 15250 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:31:52.909482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:45773 (local address 127.6.33.62:36267)
0926 16:31:52.909651 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:52.909656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:52.909959 (+ 303us) server_negotiation.cc:407] Connection header received
0926 16:31:52.910078 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:52.910082 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:52.910145 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:52.910233 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:52.910968 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:52.911790 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:52.913342 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:52.913877 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:52.913996 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:52.917758 (+ 3762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.33.2:45773: BlockingRecv error: recv got EOF from 127.6.33.2:45773 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 26 16:31:53 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.3@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:31:53 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.3@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:31:53.406500 15531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:53.398069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36267 (local address 127.6.33.3:57011)
0926 16:31:53.398695 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:53.398712 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:53.398817 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:53.399101 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:53.399104 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:53.399499 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:53.399656 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:53.399663 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:53.400611 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:53.400615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:53.401569 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:53.401580 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:53.402245 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:53.402250 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:53.402383 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:53.402965 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:53.402982 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:53.406242 (+ 3260us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.33.62:36267: Server oryx/127.6.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":385,"threads_started":1}
W20260926 16:31:53.406646 15250 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:31:53.398194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.3:57011 (local address 127.6.33.62:36267)
0926 16:31:53.398383 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:53.398388 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:53.398783 (+ 395us) server_negotiation.cc:407] Connection header received
0926 16:31:53.398918 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:53.398921 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:53.398980 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:53.399060 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:53.399778 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:53.400485 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:53.401687 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:53.402096 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:53.402187 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:53.406430 (+ 4243us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.33.3:57011: BlockingRecv error: recv got EOF from 127.6.33.3:57011 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 26 16:31:53 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.1@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:31:53 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.1@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:31:53.462509 15532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:53.454018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36267 (local address 127.6.33.1:40329)
0926 16:31:53.454581 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:53.454601 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:53.454762 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:53.455116 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:53.455121 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:53.455555 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:53.455703 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:53.455711 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:53.456814 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:53.456819 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:53.457699 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:53.457710 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:53.458457 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:53.458462 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:53.458726 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:53.459346 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:53.459365 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:53.462278 (+ 2913us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.33.62:36267: Server oryx/127.6.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"spinlock_wait_cycles":25984,"thread_start_us":440,"threads_started":1}
W20260926 16:31:53.462721 15250 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:31:53.454209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:40329 (local address 127.6.33.62:36267)
0926 16:31:53.454443 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:53.454449 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:53.454715 (+ 266us) server_negotiation.cc:407] Connection header received
0926 16:31:53.454903 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:53.454908 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:53.454976 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:53.455085 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:53.455862 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:53.456635 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:53.457857 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:53.458295 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:53.458433 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:53.462459 (+ 4026us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.33.1:40329: BlockingRecv error: recv got EOF from 127.6.33.1:40329 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 26 16:31:53 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.2@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:31:53 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.2@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:31:53.926941 15533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:53.918812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36267 (local address 127.6.33.2:59653)
0926 16:31:53.919329 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:53.919347 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:53.919514 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:53.919880 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:53.919885 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:53.920275 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:53.920420 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:53.920428 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:53.921547 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:53.921554 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:53.922431 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:53.922440 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:53.923111 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:53.923117 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:53.923283 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:53.923891 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:53.923907 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:53.926583 (+ 2676us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.33.62:36267: Server oryx/127.6.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":317,"threads_started":1}
W20260926 16:31:53.927033 15250 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:31:53.918904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:59653 (local address 127.6.33.62:36267)
0926 16:31:53.919105 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:53.919112 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:53.919447 (+ 335us) server_negotiation.cc:407] Connection header received
0926 16:31:53.919655 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:53.919659 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:53.919727 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:53.919830 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:53.920596 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:53.921414 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:53.922571 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:53.922954 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:53.923086 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:53.926802 (+ 3716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.33.2:59653: BlockingRecv error: recv got EOF from 127.6.33.2:59653 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 26 16:31:54 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.3@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:31:54 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.3@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:31:54.425949 15534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:54.407471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36267 (local address 127.6.33.3:45043)
0926 16:31:54.407944 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:54.407960 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:54.408071 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:54.408387 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:54.408390 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:54.408743 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:54.408955 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:54.408963 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:54.409931 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:54.409935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:54.410905 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:54.410916 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:54.411718 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:54.411724 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:54.411899 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:54.422530 (+ 10631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:54.422552 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:54.425601 (+ 3049us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.33.62:36267: Server oryx/127.6.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":273,"threads_started":1}
W20260926 16:31:54.426090 15250 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:31:54.407616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.3:45043 (local address 127.6.33.62:36267)
0926 16:31:54.407855 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:54.407862 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:54.408039 (+ 177us) server_negotiation.cc:407] Connection header received
0926 16:31:54.408174 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:54.408178 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:54.408239 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:54.408342 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:54.409067 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:54.409809 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:54.411046 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:54.411535 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:54.411713 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:54.425846 (+ 14133us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.33.3:45043: BlockingRecv error: recv got EOF from 127.6.33.3:45043 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Sep 26 16:31:54 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.1@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:31:54 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.1@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:31:54.471810 15535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:54.463429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36267 (local address 127.6.33.1:46789)
0926 16:31:54.464063 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:54.464079 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:54.464195 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:54.464495 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:54.464498 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:54.464901 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:54.465071 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:54.465081 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:54.466033 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:54.466037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:54.466926 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:54.466937 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:54.467588 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:54.467592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:54.467698 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:54.468349 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:54.468367 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:54.471394 (+ 3027us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.33.62:36267: Server oryx/127.6.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":271,"threads_started":1}
W20260926 16:31:54.471910 15250 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:31:54.463501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:46789 (local address 127.6.33.62:36267)
0926 16:31:54.463672 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:54.463676 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:54.464149 (+ 473us) server_negotiation.cc:407] Connection header received
0926 16:31:54.464288 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:54.464291 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:54.464372 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:54.464453 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:54.465184 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:54.465908 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:54.467043 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:54.467459 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:54.467545 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:54.471651 (+ 4106us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.33.1:46789: BlockingRecv error: recv got EOF from 127.6.33.1:46789 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 26 16:31:54 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.2@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:31:54 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.2@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:31:54.936555 15536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:54.927988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36267 (local address 127.6.33.2:46435)
0926 16:31:54.928580 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:54.928596 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:54.928709 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:54.929100 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:54.929103 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:54.929470 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:54.929655 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:54.929664 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:54.930799 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:54.930802 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:54.931778 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:54.931790 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:54.932603 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:54.932609 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:54.932803 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:54.933461 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:54.933477 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:54.936190 (+ 2713us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.33.62:36267: Server oryx/127.6.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":300,"threads_started":1}
W20260926 16:31:54.936627 15250 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:31:54.928105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:46435 (local address 127.6.33.62:36267)
0926 16:31:54.928348 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:54.928357 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:54.928685 (+ 328us) server_negotiation.cc:407] Connection header received
0926 16:31:54.928906 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:54.928911 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:54.928973 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:54.929059 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:54.929796 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:54.930661 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:54.931928 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:54.932459 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:54.932580 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:54.936385 (+ 3805us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.33.2:46435: BlockingRecv error: recv got EOF from 127.6.33.2:46435 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 26 16:31:55 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.3@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:31:55 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.3@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:31:55.435436 15537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:55.426920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36267 (local address 127.6.33.3:49239)
0926 16:31:55.427568 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:55.427583 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:55.427693 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:55.428003 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:55.428006 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:55.428420 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:55.428581 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:55.428589 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:55.429618 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:55.429622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:55.430839 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:55.430850 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:55.431763 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:55.431770 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:55.431973 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:55.432530 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:55.432547 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:55.435204 (+ 2657us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.33.62:36267: Server oryx/127.6.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":295,"threads_started":1}
W20260926 16:31:55.435623 15250 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:31:55.426975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.3:49239 (local address 127.6.33.62:36267)
0926 16:31:55.427145 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:55.427150 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:55.427656 (+ 506us) server_negotiation.cc:407] Connection header received
0926 16:31:55.427792 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:55.427795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:55.427874 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:55.427955 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:55.428698 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:55.429488 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:55.430975 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:55.431581 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:55.431719 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:55.435399 (+ 3680us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.33.3:49239: BlockingRecv error: recv got EOF from 127.6.33.3:49239 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 26 16:31:55 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.1@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:31:55 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.1@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:31:55.481463 15538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:55.472980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36267 (local address 127.6.33.1:53173)
0926 16:31:55.473494 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:55.473509 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:55.473616 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:55.473902 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:55.473905 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:55.474279 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:55.474433 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:55.474441 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:55.475432 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:55.475436 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:55.476360 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:55.476371 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:55.477117 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:55.477121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:55.477237 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:55.477838 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:55.477858 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:55.481235 (+ 3377us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.33.62:36267: Server oryx/127.6.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":285,"threads_started":1}
W20260926 16:31:55.481761 15250 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:31:55.473274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:53173 (local address 127.6.33.62:36267)
0926 16:31:55.473450 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:55.473454 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:55.473582 (+ 128us) server_negotiation.cc:407] Connection header received
0926 16:31:55.473698 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:55.473701 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:55.473761 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:55.473841 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:55.474567 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:55.475292 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:55.476504 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:55.476970 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:55.477064 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:55.481476 (+ 4412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.33.1:53173: BlockingRecv error: recv got EOF from 127.6.33.1:53173 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 26 16:31:55 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.2@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:31:55 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.2@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:31:55.946092 15539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:55.937547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36267 (local address 127.6.33.2:38957)
0926 16:31:55.938136 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:55.938161 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:55.938308 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:55.938612 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:55.938615 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:55.939018 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:55.939179 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:55.939187 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:55.940357 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:55.940362 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:55.941458 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:55.941470 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:55.942272 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:55.942277 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:55.942421 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:55.943012 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:55.943028 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:55.945865 (+ 2837us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.33.62:36267: Server oryx/127.6.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":344,"threads_started":1}
W20260926 16:31:55.946307 15250 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:31:55.937678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:38957 (local address 127.6.33.62:36267)
0926 16:31:55.937872 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:55.937877 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:55.938232 (+ 355us) server_negotiation.cc:407] Connection header received
0926 16:31:55.938424 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:55.938428 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:55.938491 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:55.938574 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:55.939298 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:55.940176 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:55.941616 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:55.942089 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:55.942197 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:55.946074 (+ 3877us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.33.2:38957: BlockingRecv error: recv got EOF from 127.6.33.2:38957 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 26 16:31:56 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.3@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:31:56 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.3@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:31:56.445282 15540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:56.436488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36267 (local address 127.6.33.3:42513)
0926 16:31:56.437030 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:56.437052 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:56.437207 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:56.437614 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:56.437617 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:56.438172 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:56.438447 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:56.438461 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:56.439683 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:56.439687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:56.440601 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:56.440612 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:56.441463 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:56.441468 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:56.441587 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:56.442320 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:56.442339 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:56.445058 (+ 2719us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.33.62:36267: Server oryx/127.6.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":298,"threads_started":1}
W20260926 16:31:56.445539 15250 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:31:56.436614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.3:42513 (local address 127.6.33.62:36267)
0926 16:31:56.436901 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:56.436908 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:56.437167 (+ 259us) server_negotiation.cc:407] Connection header received
0926 16:31:56.437351 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:56.437357 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:56.437461 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:56.437582 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:56.438634 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:56.439536 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:56.440812 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:56.441283 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:56.441392 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:56.445252 (+ 3860us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.33.3:42513: BlockingRecv error: recv got EOF from 127.6.33.3:42513 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Sep 26 16:31:56 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.1@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:31:56 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.1@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:31:56.491228 15541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:56.482520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36267 (local address 127.6.33.1:47483)
0926 16:31:56.483019 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:56.483035 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:56.483150 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:56.483479 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:56.483483 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:56.483863 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:56.484033 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:56.484040 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:56.485041 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:56.485045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:56.485956 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:56.485967 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:56.486832 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:56.486837 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:56.487027 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:56.487739 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:56.487757 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:56.490872 (+ 3115us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.33.62:36267: Server oryx/127.6.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":298,"threads_started":1}
W20260926 16:31:56.491392 15250 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:31:56.482624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:47483 (local address 127.6.33.62:36267)
0926 16:31:56.482921 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:56.482927 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:56.483120 (+ 193us) server_negotiation.cc:407] Connection header received
0926 16:31:56.483259 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:56.483263 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:56.483342 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:56.483427 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:56.484182 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:56.484924 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:56.486105 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:56.486628 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:56.486797 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:56.491113 (+ 4316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.33.1:47483: BlockingRecv error: recv got EOF from 127.6.33.1:47483 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Sep 26 16:31:56 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.2@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:31:56 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.2@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:31:56.955289 15542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:56.947101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36267 (local address 127.6.33.2:58153)
0926 16:31:56.947624 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:56.947640 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:56.947762 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:56.948049 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:56.948052 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:56.948452 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:56.948629 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:56.948639 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:56.949618 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:56.949622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:56.950566 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:56.950578 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:56.951252 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:56.951256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:56.951389 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:56.952054 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:56.952071 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:56.954977 (+ 2906us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.33.62:36267: Server oryx/127.6.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":298,"threads_started":1}
W20260926 16:31:56.955560 15250 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:31:56.947162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:58153 (local address 127.6.33.62:36267)
0926 16:31:56.947350 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:56.947354 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:56.947713 (+ 359us) server_negotiation.cc:407] Connection header received
0926 16:31:56.947862 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:56.947865 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:56.947927 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:56.948010 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:56.948775 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:56.949507 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:56.950696 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:56.951118 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:56.951209 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:56.955235 (+ 4026us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.33.2:58153: BlockingRecv error: recv got EOF from 127.6.33.2:58153 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 26 16:31:57 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.3@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:31:57 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.3@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:31:57.454922 15543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:57.446348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36267 (local address 127.6.33.3:50393)
0926 16:31:57.446904 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:57.446920 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:57.447086 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:57.447447 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:57.447453 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:57.447867 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:57.448029 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:57.448038 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:57.449168 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:57.449175 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:57.450129 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:57.450141 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:57.450864 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:57.450870 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:57.451054 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:57.451686 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:57.451704 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:57.454568 (+ 2864us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.33.62:36267: Server oryx/127.6.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":295,"threads_started":1}
W20260926 16:31:57.455039 15250 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:31:57.446503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.3:50393 (local address 127.6.33.62:36267)
0926 16:31:57.446705 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:57.446714 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:57.447018 (+ 304us) server_negotiation.cc:407] Connection header received
0926 16:31:57.447224 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:57.447229 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:57.447305 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:57.447407 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:57.448177 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:57.449037 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:57.450281 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:57.450698 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:57.450836 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:57.454806 (+ 3970us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.33.3:50393: BlockingRecv error: recv got EOF from 127.6.33.3:50393 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 26 16:31:57 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.1@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:31:57 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.1@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:31:57.501072 15544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:57.492271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36267 (local address 127.6.33.1:48103)
0926 16:31:57.492830 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:57.492849 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:57.492975 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:57.493383 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:57.493386 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:57.493733 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:57.493878 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:57.493886 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:57.494957 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:57.494960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:57.495968 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:57.495983 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:57.497059 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:57.497065 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:57.497252 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:57.498001 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:57.498017 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:57.500846 (+ 2829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.33.62:36267: Server oryx/127.6.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":320,"threads_started":1}
W20260926 16:31:57.501329 15250 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:31:57.492428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:48103 (local address 127.6.33.62:36267)
0926 16:31:57.492611 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:57.492617 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:57.492944 (+ 327us) server_negotiation.cc:407] Connection header received
0926 16:31:57.493078 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:57.493082 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:57.493212 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:57.493311 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:57.494026 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:57.494841 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:57.496121 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:57.496871 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:57.497000 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:57.501057 (+ 4057us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.33.1:48103: BlockingRecv error: recv got EOF from 127.6.33.1:48103 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 26 16:31:57 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.2@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:31:57 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.2@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:31:57.965173 15545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:57.956444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36267 (local address 127.6.33.2:56643)
0926 16:31:57.957015 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:57.957040 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:57.957158 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:57.957448 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:57.957451 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:57.957803 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:57.957939 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:57.957947 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:57.959380 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:57.959386 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:57.960645 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:57.960658 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:57.961394 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:57.961401 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:57.961567 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:57.962210 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:57.962227 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:57.964947 (+ 2720us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.33.62:36267: Server oryx/127.6.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":364,"threads_started":1}
W20260926 16:31:57.965386 15250 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:31:57.956516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:56643 (local address 127.6.33.62:36267)
0926 16:31:57.956736 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:57.956743 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:57.957118 (+ 375us) server_negotiation.cc:407] Connection header received
0926 16:31:57.957258 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:57.957262 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:57.957326 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:57.957414 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:57.958096 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:57.959239 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:57.960786 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:57.961204 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:57.961294 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:57.965136 (+ 3842us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.33.2:56643: BlockingRecv error: recv got EOF from 127.6.33.2:56643 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 26 16:31:58 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.3@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:31:58 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.3@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:31:58.465379 15546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:58.456088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36267 (local address 127.6.33.3:37741)
0926 16:31:58.456672 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:58.456689 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:58.456869 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:58.457224 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:58.457228 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:58.457634 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:58.457771 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:58.457779 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:58.459422 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:58.459429 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:58.460402 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:58.460415 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:58.461185 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:58.461191 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:58.461380 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:58.461993 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:58.462022 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:58.465139 (+ 3117us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.33.62:36267: Server oryx/127.6.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":359,"threads_started":1}
W20260926 16:31:58.465603 15250 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:31:58.456386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.3:37741 (local address 127.6.33.62:36267)
0926 16:31:58.456615 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:58.456620 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:58.456819 (+ 199us) server_negotiation.cc:407] Connection header received
0926 16:31:58.456992 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:58.456997 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:58.457063 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:58.457158 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:58.457886 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:58.459272 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:58.460564 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:58.461020 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:58.461160 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:58.465349 (+ 4189us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.33.3:37741: BlockingRecv error: recv got EOF from 127.6.33.3:37741 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Sep 26 16:31:58 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.1@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:31:58 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.1@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:31:58.510991 15547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:58.502078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36267 (local address 127.6.33.1:39607)
0926 16:31:58.502600 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:58.502614 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:58.502763 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:58.503126 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:58.503131 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:58.503552 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:58.503713 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:58.503721 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:58.504696 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:58.504703 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:58.505710 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:58.505722 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:58.506600 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:58.506608 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:58.506794 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:58.507498 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:58.507525 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:58.510705 (+ 3180us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.33.62:36267: Server oryx/127.6.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":330,"threads_started":1}
W20260926 16:31:58.511083 15250 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:31:58.502144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:39607 (local address 127.6.33.62:36267)
0926 16:31:58.502372 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:58.502381 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:58.502694 (+ 313us) server_negotiation.cc:407] Connection header received
0926 16:31:58.502905 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:58.502920 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:58.502994 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:58.503096 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:58.503860 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:58.504567 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:58.505851 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:58.506367 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:58.506515 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:58.510900 (+ 4385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.33.1:39607: BlockingRecv error: recv got EOF from 127.6.33.1:39607 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 26 16:31:58 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.2@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:31:58 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.2@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:31:58.975500 15548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:58.966330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36267 (local address 127.6.33.2:51157)
0926 16:31:58.966853 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:58.966873 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:58.967003 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:58.967361 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:58.967366 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:58.967815 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:58.968042 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:58.968055 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:58.969320 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:58.969324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:58.970350 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:58.970362 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:58.971221 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:58.971227 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:58.971357 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:58.972080 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:58.972105 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:58.975207 (+ 3102us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.33.62:36267: Server oryx/127.6.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":276,"threads_started":1}
W20260926 16:31:58.975667 15250 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:31:58.966423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:51157 (local address 127.6.33.62:36267)
0926 16:31:58.966666 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:58.966673 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:58.966998 (+ 325us) server_negotiation.cc:407] Connection header received
0926 16:31:58.967099 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:58.967104 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:58.967200 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:58.967300 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:58.968274 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:58.969159 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:58.970496 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:58.971048 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:58.971159 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:58.975441 (+ 4282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.33.2:51157: BlockingRecv error: recv got EOF from 127.6.33.2:51157 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 26 16:31:59 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.3@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:31:59 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.3@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:31:59.480243 15549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:59.466636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36267 (local address 127.6.33.3:58215)
0926 16:31:59.467229 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:59.467246 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:59.467443 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:59.467800 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:59.467805 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:59.468229 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:59.468405 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:59.468418 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:59.469494 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:59.469502 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:59.470493 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:59.470505 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:59.471227 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:59.471233 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:59.471407 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:59.476427 (+ 5020us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:59.476450 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:59.479980 (+ 3530us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.33.62:36267: Server oryx/127.6.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":323,"threads_started":1}
W20260926 16:31:59.480567 15250 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:31:59.466844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.3:58215 (local address 127.6.33.62:36267)
0926 16:31:59.467103 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:59.467110 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:59.467367 (+ 257us) server_negotiation.cc:407] Connection header received
0926 16:31:59.467587 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:59.467593 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:59.467662 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:59.467769 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:59.468549 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:59.469369 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:59.470639 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:59.471073 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:59.471212 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:59.480258 (+ 9046us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.33.3:58215: BlockingRecv error: recv got EOF from 127.6.33.3:58215 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Sep 26 16:31:59 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.1@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:31:59 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.1@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:31:59.520690 15550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:59.512070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36267 (local address 127.6.33.1:34925)
0926 16:31:59.512583 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:59.512599 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:59.512727 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:59.513082 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:59.513085 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:59.513511 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:59.513678 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:59.513687 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:59.514748 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:59.514754 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:59.515733 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:59.515746 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:59.516499 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:59.516504 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:59.516639 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:59.517342 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:59.517360 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:59.520345 (+ 2985us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.33.62:36267: Server oryx/127.6.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":292,"threads_started":1}
W20260926 16:31:59.520825 15250 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:31:59.512123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:34925 (local address 127.6.33.62:36267)
0926 16:31:59.512294 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:59.512299 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:59.512695 (+ 396us) server_negotiation.cc:407] Connection header received
0926 16:31:59.512879 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:59.512882 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:59.512958 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:59.513059 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:59.513837 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:59.514586 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:59.515872 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:59.516360 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:59.516467 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:59.520554 (+ 4087us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.33.1:34925: BlockingRecv error: recv got EOF from 127.6.33.1:34925 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 26 16:31:59 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.2@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:31:59 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.2@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:31:59.984664 15551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:31:59.976470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36267 (local address 127.6.33.2:36611)
0926 16:31:59.977007 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0926 16:31:59.977023 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:31:59.977137 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:31:59.977449 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:31:59.977453 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:31:59.977839 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:31:59.977987 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:59.977995 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:59.978975 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:59.978979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:59.979946 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:31:59.979958 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:31:59.980608 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:31:59.980612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:31:59.980824 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:59.981413 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:31:59.981431 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:31:59.984291 (+ 2860us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.33.62:36267: Server oryx/127.6.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":326,"threads_started":1}
W20260926 16:31:59.984704 15250 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:31:59.976521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:36611 (local address 127.6.33.62:36267)
0926 16:31:59.976789 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0926 16:31:59.976798 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:31:59.977103 (+ 305us) server_negotiation.cc:407] Connection header received
0926 16:31:59.977239 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:31:59.977243 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:31:59.977303 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:31:59.977390 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:31:59.978117 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:59.978837 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:59.980065 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:31:59.980475 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:31:59.980598 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:31:59.984503 (+ 3905us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.33.2:36611: BlockingRecv error: recv got EOF from 127.6.33.2:36611 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 26 16:32:00 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.3@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:32:00 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.3@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:32:00.489862 15552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:32:00.481340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36267 (local address 127.6.33.3:47619)
0926 16:32:00.482065 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0926 16:32:00.482092 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 16:32:00.482201 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:32:00.482562 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:32:00.482565 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:32:00.482932 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:32:00.483080 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:32:00.483087 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:32:00.484077 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:32:00.484081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:32:00.485154 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:32:00.485172 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:32:00.485816 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:32:00.485822 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:32:00.485946 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:32:00.486602 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:32:00.486619 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:32:00.489574 (+ 2955us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.33.62:36267: Server oryx/127.6.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":432,"threads_started":1}
W20260926 16:32:00.490151 15250 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:32:00.481430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.3:47619 (local address 127.6.33.62:36267)
0926 16:32:00.481668 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0926 16:32:00.481677 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:32:00.482149 (+ 472us) server_negotiation.cc:407] Connection header received
0926 16:32:00.482348 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:32:00.482352 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:32:00.482433 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:32:00.482521 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:32:00.483233 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:32:00.483949 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:32:00.485284 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:32:00.485694 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:32:00.485787 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:32:00.489805 (+ 4018us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.33.3:47619: BlockingRecv error: recv got EOF from 127.6.33.3:47619 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 26 16:32:00 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.1@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:32:00 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.1@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:32:00.530246 15553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:32:00.521719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36267 (local address 127.6.33.1:35947)
0926 16:32:00.522254 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0926 16:32:00.522269 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 16:32:00.522383 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:32:00.522698 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:32:00.522701 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:32:00.523096 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:32:00.523247 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:32:00.523255 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:32:00.524359 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:32:00.524363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:32:00.525293 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:32:00.525305 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:32:00.526055 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:32:00.526060 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:32:00.526174 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:32:00.526776 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:32:00.526793 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:32:00.529892 (+ 3099us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.33.62:36267: Server oryx/127.6.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":294,"threads_started":1}
W20260926 16:32:00.530563 15250 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:32:00.521864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:35947 (local address 127.6.33.62:36267)
0926 16:32:00.522032 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 16:32:00.522036 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:32:00.522365 (+ 329us) server_negotiation.cc:407] Connection header received
0926 16:32:00.522480 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:32:00.522484 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:32:00.522564 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:32:00.522662 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:32:00.523398 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:32:00.524219 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:32:00.525441 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:32:00.525886 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:32:00.525994 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:32:00.530068 (+ 4074us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.33.1:35947: BlockingRecv error: recv got EOF from 127.6.33.1:35947 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 26 16:32:00 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.2@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:32:00 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.2@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:32:00.994532 15554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:32:00.985792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36267 (local address 127.6.33.2:42551)
0926 16:32:00.986483 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0926 16:32:00.986500 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 16:32:00.986648 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:32:00.987086 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:32:00.987089 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:32:00.987530 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:32:00.987687 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:32:00.987696 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:32:00.988695 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:32:00.988699 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:32:00.989665 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:32:00.989676 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:32:00.990463 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:32:00.990469 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:32:00.990592 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:32:00.991219 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:32:00.991237 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:32:00.994185 (+ 2948us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.33.62:36267: Server oryx/127.6.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":351,"threads_started":1}
W20260926 16:32:00.994627 15250 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:32:00.986039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:42551 (local address 127.6.33.62:36267)
0926 16:32:00.986236 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 16:32:00.986242 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:32:00.986585 (+ 343us) server_negotiation.cc:407] Connection header received
0926 16:32:00.986797 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:32:00.986819 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:32:00.986909 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:32:00.987023 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:32:00.987835 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:32:00.988561 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:32:00.989790 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:32:00.990267 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:32:00.990407 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:32:00.994391 (+ 3984us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.33.2:42551: BlockingRecv error: recv got EOF from 127.6.33.2:42551 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 26 16:32:01 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.3@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:32:01 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.3@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:32:01.500525 15555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:32:01.490838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36267 (local address 127.6.33.3:41109)
0926 16:32:01.491509 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0926 16:32:01.491528 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 16:32:01.491651 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:32:01.492048 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:32:01.492052 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:32:01.492521 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:32:01.492712 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:32:01.492722 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:32:01.493824 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:32:01.493828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:32:01.495295 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:32:01.495313 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:32:01.496324 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:32:01.496332 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:32:01.496526 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:32:01.497272 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:32:01.497291 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:32:01.500253 (+ 2962us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.33.62:36267: Server oryx/127.6.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":393,"threads_started":1}
W20260926 16:32:01.500687 15250 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:32:01.490997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.3:41109 (local address 127.6.33.62:36267)
0926 16:32:01.491173 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 16:32:01.491178 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:32:01.491607 (+ 429us) server_negotiation.cc:407] Connection header received
0926 16:32:01.491791 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:32:01.491795 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:32:01.491880 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:32:01.491970 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:32:01.492929 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:32:01.493654 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:32:01.495451 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:32:01.496076 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:32:01.496213 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:32:01.500461 (+ 4248us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.33.3:41109: BlockingRecv error: recv got EOF from 127.6.33.3:41109 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 26 16:32:01 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.1@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:32:01 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.1@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 16:32:01.541247 15250 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:32:01.531697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:52011 (local address 127.6.33.62:36267)
0926 16:32:01.532021 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0926 16:32:01.532027 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:32:01.532133 (+ 106us) server_negotiation.cc:407] Connection header received
0926 16:32:01.532408 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:32:01.532414 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:32:01.532522 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:32:01.532665 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:32:01.533695 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:32:01.534514 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:32:01.535877 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:32:01.536369 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:32:01.536521 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:32:01.540968 (+ 4447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.33.1:52011: BlockingRecv error: recv got EOF from 127.6.33.1:52011 (error 108)
Metrics: {"server-negotiator.queue_time_us":196}
I20260926 16:32:01.541241 15556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:32:01.531521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36267 (local address 127.6.33.1:52011)
0926 16:32:01.532043 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0926 16:32:01.532065 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 16:32:01.532213 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:32:01.532705 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:32:01.532712 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:32:01.533265 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:32:01.533537 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:32:01.533548 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:32:01.534681 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:32:01.534687 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:32:01.535717 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:32:01.535729 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:32:01.536536 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:32:01.536542 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:32:01.536780 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:32:01.537552 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:32:01.537587 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:32:01.540728 (+ 3141us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.33.62:36267: Server oryx/127.6.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":290,"threads_started":1}
Sep 26 16:32:01 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.2@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:32:01 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.2@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:32:02.005144 15557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:32:01.995635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36267 (local address 127.6.33.2:44057)
0926 16:32:01.996135 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0926 16:32:01.996153 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 16:32:01.996268 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:32:01.996569 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:32:01.996573 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:32:01.997021 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:32:01.997244 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:32:01.997255 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:32:01.998516 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:32:01.998523 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:32:01.999974 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:32:01.999988 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:32:02.000888 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:32:02.000893 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:32:02.001020 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:32:02.001649 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:32:02.001676 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:32:02.004863 (+ 3187us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.33.62:36267: Server oryx/127.6.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":269,"threads_started":1}
W20260926 16:32:02.005365 15250 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:32:01.995758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:44057 (local address 127.6.33.62:36267)
0926 16:32:01.995995 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0926 16:32:01.996000 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:32:01.996230 (+ 230us) server_negotiation.cc:407] Connection header received
0926 16:32:01.996366 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:32:01.996370 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:32:01.996432 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:32:01.996523 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:32:01.997427 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:32:01.998383 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:32:02.000137 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:32:02.000630 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:32:02.000733 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:32:02.005105 (+ 4372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.33.2:44057: BlockingRecv error: recv got EOF from 127.6.33.2:44057 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Sep 26 16:32:02 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.3@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:32:02 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.3@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:32:02.510448 15558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:32:02.501665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36267 (local address 127.6.33.3:40659)
0926 16:32:02.502152 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0926 16:32:02.502167 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 16:32:02.502318 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:32:02.502715 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:32:02.502718 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:32:02.503071 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:32:02.503243 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:32:02.503253 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:32:02.504336 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:32:02.504340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:32:02.505340 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:32:02.505353 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:32:02.506123 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:32:02.506128 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:32:02.506249 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:32:02.506879 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:32:02.506896 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:32:02.510110 (+ 3214us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.33.62:36267: Server oryx/127.6.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":259,"threads_started":1}
W20260926 16:32:02.510592 15250 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:32:02.501718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.3:40659 (local address 127.6.33.62:36267)
0926 16:32:02.501948 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 16:32:02.501956 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:32:02.502252 (+ 296us) server_negotiation.cc:407] Connection header received
0926 16:32:02.502446 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:32:02.502451 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:32:02.502522 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:32:02.502645 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:32:02.503409 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:32:02.504171 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:32:02.505518 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:32:02.505986 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:32:02.506091 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:32:02.510356 (+ 4265us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.33.3:40659: BlockingRecv error: recv got EOF from 127.6.33.3:40659 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 26 16:32:02 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.1@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:32:02 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.1@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:32:02.551784 15559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:32:02.542555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36267 (local address 127.6.33.1:33381)
0926 16:32:02.543105 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0926 16:32:02.543121 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:32:02.543286 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:32:02.543678 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:32:02.543683 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:32:02.544121 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:32:02.544318 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:32:02.544326 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:32:02.545457 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:32:02.545463 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:32:02.546412 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:32:02.546423 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:32:02.547161 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:32:02.547167 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:32:02.547351 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:32:02.548002 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:32:02.548021 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:32:02.551381 (+ 3360us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.33.62:36267: Server oryx/127.6.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":303,"threads_started":1}
W20260926 16:32:02.551888 15250 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:32:02.542688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:33381 (local address 127.6.33.62:36267)
0926 16:32:02.542945 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0926 16:32:02.542953 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:32:02.543249 (+ 296us) server_negotiation.cc:407] Connection header received
0926 16:32:02.543425 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:32:02.543434 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:32:02.543507 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:32:02.543619 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:32:02.544498 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:32:02.545308 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:32:02.546561 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:32:02.547005 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:32:02.547136 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:32:02.551673 (+ 4537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.33.1:33381: BlockingRecv error: recv got EOF from 127.6.33.1:33381 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 26 16:32:03 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.2@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:32:03 dist-test-slave-khd3 krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.33.2@KRBTEST.COM for oryx/127.6.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:32:03.015528 15560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:32:03.006413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36267 (local address 127.6.33.2:54103)
0926 16:32:03.006958 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0926 16:32:03.006975 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 16:32:03.007100 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:32:03.007468 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:32:03.007471 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:32:03.007868 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:32:03.008051 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:32:03.008060 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:32:03.009302 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:32:03.009307 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:32:03.010260 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:32:03.010275 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:32:03.011209 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:32:03.011214 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:32:03.011375 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:32:03.012109 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:32:03.012140 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:32:03.015263 (+ 3123us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.33.62:36267: Server oryx/127.6.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":315,"threads_started":1}
W20260926 16:32:03.015729 15250 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:32:03.006559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:54103 (local address 127.6.33.62:36267)
0926 16:32:03.006763 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 16:32:03.006768 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:32:03.007079 (+ 311us) server_negotiation.cc:407] Connection header received
0926 16:32:03.007263 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:32:03.007267 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:32:03.007331 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:32:03.007432 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:32:03.008279 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:32:03.009109 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:32:03.010422 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:32:03.010984 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:32:03.011105 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:32:03.015435 (+ 4330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.33.2:54103: BlockingRecv error: recv got EOF from 127.6.33.2:54103 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 16:32:03.335448 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 15101
I20260926 16:32:03.347970 15562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:32:03.336577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34332 (local address 127.6.33.1:40909)
0926 16:32:03.337166 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0926 16:32:03.337174 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:32:03.337201 (+ 27us) server_negotiation.cc:407] Connection header received
0926 16:32:03.337277 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:32:03.337288 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:32:03.337539 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:32:03.337708 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:32:03.339312 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:32:03.340405 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:32:03.341703 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:32:03.342284 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:32:03.342392 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:32:03.344928 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:32:03.344937 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:32:03.344952 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:32:03.344989 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:32:03.347483 (+ 2494us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.33.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 16:32:03.347741 (+ 258us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34332: Request ticket server kudu/127.6.33.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":280,"threads_started":1}
W20260926 16:32:03.348093 15562 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34332: Request ticket server kudu/127.6.33.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260926 16:32:03.348275 6276 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 15101: Not authorized: Client connection negotiation failed: client connection to 127.6.33.1:40909: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.33.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 16:32:03.348500 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 15101
I20260926 16:32:03.365294 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 15252
I20260926 16:32:03.378866 15563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:32:03.366477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49966 (local address 127.6.33.2:43561)
0926 16:32:03.367022 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0926 16:32:03.367039 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 16:32:03.367070 (+ 31us) server_negotiation.cc:407] Connection header received
0926 16:32:03.367129 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:32:03.367135 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:32:03.367404 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:32:03.367570 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:32:03.369003 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:32:03.370207 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:32:03.371730 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:32:03.372851 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:32:03.372980 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:32:03.375581 (+ 2601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:32:03.375590 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:32:03.375604 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:32:03.375638 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:32:03.378357 (+ 2719us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.33.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 16:32:03.378619 (+ 262us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49966: Request ticket server kudu/127.6.33.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":250,"threads_started":1}
W20260926 16:32:03.378997 15563 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49966: Request ticket server kudu/127.6.33.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260926 16:32:03.379065 6276 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 15252: Not authorized: Client connection negotiation failed: client connection to 127.6.33.2:43561: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.33.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 16:32:03.379299 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 15252
I20260926 16:32:03.395846 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 15384
I20260926 16:32:03.408576 15564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:32:03.396889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53794 (local address 127.6.33.3:42631)
0926 16:32:03.397525 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0926 16:32:03.397533 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:32:03.397559 (+ 26us) server_negotiation.cc:407] Connection header received
0926 16:32:03.397619 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:32:03.397625 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:32:03.397877 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:32:03.398027 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:32:03.399318 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:32:03.400379 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:32:03.401864 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:32:03.402446 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:32:03.402551 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:32:03.405307 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:32:03.405317 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:32:03.405332 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:32:03.405378 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:32:03.408062 (+ 2684us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.33.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 16:32:03.408337 (+ 275us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53794: Request ticket server kudu/127.6.33.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":334,"threads_started":1}
W20260926 16:32:03.408715 15564 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53794: Request ticket server kudu/127.6.33.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260926 16:32:03.408813 6276 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 15384: Not authorized: Client connection negotiation failed: client connection to 127.6.33.3:42631: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.33.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 16:32:03.408968 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 15384
I20260926 16:32:03.425293 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 15032
I20260926 16:32:03.426613 15094 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:32:03.543381 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 15032
2026-09-26T16:32:03Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21183 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-26T16:32:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:32:03Z Disabled control of system clock
I20260926 16:32:03.624557 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440233850748-6276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440233850748-6276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440233850748-6276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:43141
--webserver_interface=127.6.33.62
--webserver_port=0
--builtin_ntp_servers=127.6.33.20:36619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.33.62:43141
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 16:32:03.955983 15569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:32:03.956303 15569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:32:03.956441 15569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:32:03.964120 15569 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:32:03.964254 15569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:32:03.964327 15569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:32:03.964443 15569 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:32:03.964514 15569 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:32:03.974177 15569 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:36619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440233850748-6276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440233850748-6276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.33.62:43141
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:43141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440233850748-6276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:32:03.975564 15569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:32:03.977541 15569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:32:03.989593 15577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:32:03.989717 15575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:32:03.989946 15574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:32:03.990686 15569 server_base.cc:1034] running on GCE node
I20260926 16:32:03.991591 15569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:32:03.993136 15569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:32:03.994375 15569 hybrid_clock.cc:648] HybridClock initialized: now 1790440323994323 us; error 47 us; skew 500 ppm
I20260926 16:32:03.994750 15569 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:32:03.997056 15569 webserver.cc:460] Webserver started at http://127.6.33.62:41039/ using document root <none> and password file <none>
I20260926 16:32:03.997773 15569 fs_manager.cc:359] Metadata directory not provided
I20260926 16:32:03.997903 15569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:32:03.998252 15569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:32:04.001210 15569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440233850748-6276-0/minicluster-data/master-0/data/instance:
uuid: "51aacf6d209946908c90f79ce15949f7"
format_stamp: "Formatted at 2026-09-26 16:32:03 on dist-test-slave-khd3"
server_key: "50b4a76e3396bbd06cef7eb39d231914"
server_key_iv: "59c453f5e1ed6b8dd0fe3af18228f7ec"
server_key_version: "encryptionkey@0"
I20260926 16:32:04.001976 15569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440233850748-6276-0/minicluster-data/master-0/wal/instance:
uuid: "51aacf6d209946908c90f79ce15949f7"
format_stamp: "Formatted at 2026-09-26 16:32:03 on dist-test-slave-khd3"
server_key: "50b4a76e3396bbd06cef7eb39d231914"
server_key_iv: "59c453f5e1ed6b8dd0fe3af18228f7ec"
server_key_version: "encryptionkey@0"
I20260926 16:32:04.007428 15569 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260926 16:32:04.011245 15583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:32:04.012885 15569 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:32:04.013078 15569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440233850748-6276-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440233850748-6276-0/minicluster-data/master-0/wal
uuid: "51aacf6d209946908c90f79ce15949f7"
format_stamp: "Formatted at 2026-09-26 16:32:03 on dist-test-slave-khd3"
server_key: "50b4a76e3396bbd06cef7eb39d231914"
server_key_iv: "59c453f5e1ed6b8dd0fe3af18228f7ec"
server_key_version: "encryptionkey@0"
I20260926 16:32:04.013319 15569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440233850748-6276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440233850748-6276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440233850748-6276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:32:04.025877 15569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:32:04.026783 15569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:32:04.027091 15569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:32:04.046964 15569 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.62:43141
I20260926 16:32:04.046974 15634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.62:43141 every 8 connection(s)
I20260926 16:32:04.048462 15569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
I20260926 16:32:04.052982 15635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:32:04.058113 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 15569
I20260926 16:32:04.058347 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440233850748-6276-0/minicluster-data/master-0/wal/instance
I20260926 16:32:04.059166 6276 external_mini_cluster.cc:1428] Setting key 7a9e8d4419bc91fa46c55499b709333e
I20260926 16:32:04.065413 15635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51aacf6d209946908c90f79ce15949f7: Bootstrap starting.
I20260926 16:32:04.069058 15635 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51aacf6d209946908c90f79ce15949f7: Neither blocks nor log segments found. Creating new log.
I20260926 16:32:04.070243 15635 log.cc:824] T 00000000000000000000000000000000 P 51aacf6d209946908c90f79ce15949f7: Log is configured to *not* fsync() on all Append() calls
I20260926 16:32:04.073578 15635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51aacf6d209946908c90f79ce15949f7: No bootstrap required, opened a new log
I20260926 16:32:04.074460 15638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:32:04.061563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47598 (local address 127.6.33.62:43141)
0926 16:32:04.062736 (+ 1173us) server_negotiation.cc:206] Beginning negotiation
0926 16:32:04.062749 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 16:32:04.062803 (+ 54us) server_negotiation.cc:407] Connection header received
0926 16:32:04.064646 (+ 1843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:32:04.064671 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:32:04.065257 (+ 586us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:32:04.065697 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:32:04.066321 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:32:04.067780 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:32:04.069784 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:32:04.070602 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:32:04.070811 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:32:04.072055 (+ 1244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:32:04.072062 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:32:04.072076 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:32:04.072605 (+ 529us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:32:04.072704 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:32:04.072724 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:32:04.072891 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 16:32:04.073074 (+ 183us) server_negotiation.cc:299] Negotiation successful
0926 16:32:04.073451 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":413,"threads_started":1}
I20260926 16:32:04.080943 15635 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51aacf6d209946908c90f79ce15949f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51aacf6d209946908c90f79ce15949f7" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 43141 } }
I20260926 16:32:04.081301 15635 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51aacf6d209946908c90f79ce15949f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:32:04.081427 15635 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51aacf6d209946908c90f79ce15949f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51aacf6d209946908c90f79ce15949f7, State: Initialized, Role: FOLLOWER
I20260926 16:32:04.082013 15635 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51aacf6d209946908c90f79ce15949f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51aacf6d209946908c90f79ce15949f7" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 43141 } }
I20260926 16:32:04.082263 15635 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51aacf6d209946908c90f79ce15949f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:32:04.082413 15635 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51aacf6d209946908c90f79ce15949f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:32:04.082589 15635 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51aacf6d209946908c90f79ce15949f7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:32:04.084857 15635 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51aacf6d209946908c90f79ce15949f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51aacf6d209946908c90f79ce15949f7" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 43141 } }
I20260926 16:32:04.085354 15635 leader_election.cc:302] T 00000000000000000000000000000000 P 51aacf6d209946908c90f79ce15949f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51aacf6d209946908c90f79ce15949f7; no voters:
I20260926 16:32:04.085932 15635 leader_election.cc:288] T 00000000000000000000000000000000 P 51aacf6d209946908c90f79ce15949f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:32:04.086084 15640 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51aacf6d209946908c90f79ce15949f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:32:04.086796 15640 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51aacf6d209946908c90f79ce15949f7 [term 1 LEADER]: Becoming Leader. State: Replica: 51aacf6d209946908c90f79ce15949f7, State: Running, Role: LEADER
I20260926 16:32:04.087345 15640 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51aacf6d209946908c90f79ce15949f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51aacf6d209946908c90f79ce15949f7" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 43141 } }
I20260926 16:32:04.088037 15635 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51aacf6d209946908c90f79ce15949f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:32:04.090046 15641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51aacf6d209946908c90f79ce15949f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "51aacf6d209946908c90f79ce15949f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51aacf6d209946908c90f79ce15949f7" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 43141 } } }
I20260926 16:32:04.090469 15641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51aacf6d209946908c90f79ce15949f7 [sys.catalog]: This master's current role is: LEADER
I20260926 16:32:04.090148 15642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51aacf6d209946908c90f79ce15949f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51aacf6d209946908c90f79ce15949f7. Latest consensus state: current_term: 1 leader_uuid: "51aacf6d209946908c90f79ce15949f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51aacf6d209946908c90f79ce15949f7" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 43141 } } }
I20260926 16:32:04.090957 15642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51aacf6d209946908c90f79ce15949f7 [sys.catalog]: This master's current role is: LEADER
I20260926 16:32:04.092692 15647 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:32:04.098032 15647 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:32:04.108090 15647 catalog_manager.cc:1347] Generated new cluster ID: 2b0d2fbb84064b79bea305145da79c38
I20260926 16:32:04.108208 15647 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:32:04.128058 15647 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:32:04.129468 15647 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:32:04.136045 15647 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 51aacf6d209946908c90f79ce15949f7: Generated new TSK 0
I20260926 16:32:04.136996 15647 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:32:04.149986 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.1:0
--local_ip_for_outbound_sockets=127.6.33.1
--webserver_interface=127.6.33.1
--webserver_port=0
--tserver_master_addrs=127.6.33.62:43141
--builtin_ntp_servers=127.6.33.20:36619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 16:32:04.487507 15659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:32:04.487870 15659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:32:04.488020 15659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:32:04.495926 15659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:32:04.496083 15659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:32:04.496275 15659 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.1
I20260926 16:32:04.505816 15659 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:36619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.33.1
--webserver_port=0
--tserver_master_addrs=127.6.33.62:43141
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.1
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:32:04.507246 15659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:32:04.509099 15659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:32:04.522861 15664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:32:04.522817 15667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:32:04.522893 15665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:32:04.523722 15659 server_base.cc:1034] running on GCE node
I20260926 16:32:04.524400 15659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:32:04.525439 15659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:32:04.526683 15659 hybrid_clock.cc:648] HybridClock initialized: now 1790440324526612 us; error 62 us; skew 500 ppm
I20260926 16:32:04.527036 15659 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:32:04.539048 15659 webserver.cc:460] Webserver started at http://127.6.33.1:36637/ using document root <none> and password file <none>
I20260926 16:32:04.539878 15659 fs_manager.cc:359] Metadata directory not provided
I20260926 16:32:04.540031 15659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:32:04.540387 15659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:32:04.543627 15659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-0/data/instance:
uuid: "eeb85b519acf497fa536cda8be702b56"
format_stamp: "Formatted at 2026-09-26 16:32:04 on dist-test-slave-khd3"
server_key: "a7a68487c06a688a1c3600a95e6d70fb"
server_key_iv: "a5f6b6feb2755acf23f9813e77d64d7f"
server_key_version: "encryptionkey@0"
I20260926 16:32:04.544477 15659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-0/wal/instance:
uuid: "eeb85b519acf497fa536cda8be702b56"
format_stamp: "Formatted at 2026-09-26 16:32:04 on dist-test-slave-khd3"
server_key: "a7a68487c06a688a1c3600a95e6d70fb"
server_key_iv: "a5f6b6feb2755acf23f9813e77d64d7f"
server_key_version: "encryptionkey@0"
I20260926 16:32:04.550033 15659 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260926 16:32:04.553911 15673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:32:04.555348 15659 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 16:32:04.555545 15659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-0/wal
uuid: "eeb85b519acf497fa536cda8be702b56"
format_stamp: "Formatted at 2026-09-26 16:32:04 on dist-test-slave-khd3"
server_key: "a7a68487c06a688a1c3600a95e6d70fb"
server_key_iv: "a5f6b6feb2755acf23f9813e77d64d7f"
server_key_version: "encryptionkey@0"
I20260926 16:32:04.555802 15659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:32:04.581090 15659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:32:04.582006 15659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:32:04.582304 15659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:32:04.583431 15659 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:32:04.585151 15659 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:32:04.585278 15659 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:32:04.585410 15659 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:32:04.585496 15659 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:32:04.620152 15659 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.1:46373
I20260926 16:32:04.620271 15785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.1:46373 every 8 connection(s)
I20260926 16:32:04.621583 15659 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
I20260926 16:32:04.624688 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 15659
I20260926 16:32:04.624936 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-0/wal/instance
I20260926 16:32:04.625597 6276 external_mini_cluster.cc:1428] Setting key 8d8caeadea4042a0361c2a8374475ad1
I20260926 16:32:04.628721 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.2:0
--local_ip_for_outbound_sockets=127.6.33.2
--webserver_interface=127.6.33.2
--webserver_port=0
--tserver_master_addrs=127.6.33.62:43141
--builtin_ntp_servers=127.6.33.20:36619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 16:32:04.637185 15788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:32:04.624512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:38527 (local address 127.6.33.62:43141)
0926 16:32:04.625025 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0926 16:32:04.625031 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:32:04.626562 (+ 1531us) server_negotiation.cc:407] Connection header received
0926 16:32:04.628855 (+ 2293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:32:04.628861 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:32:04.628963 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:32:04.629068 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:32:04.630596 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:32:04.631361 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:32:04.633951 (+ 2590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:32:04.634367 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:32:04.634475 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:32:04.636292 (+ 1817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:32:04.636297 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:32:04.636300 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:32:04.636321 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:32:04.636378 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:32:04.636389 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:32:04.636487 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 16:32:04.636714 (+ 227us) server_negotiation.cc:299] Negotiation successful
0926 16:32:04.636842 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":276,"threads_started":1}
I20260926 16:32:04.638432 15789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:32:04.625227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:43141 (local address 127.6.33.1:38527)
0926 16:32:04.626343 (+ 1116us) negotiation.cc:107] Waiting for socket to connect
0926 16:32:04.626379 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 16:32:04.628462 (+ 2083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:32:04.629348 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:32:04.629363 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:32:04.629511 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:32:04.630400 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:32:04.630429 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:32:04.631532 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:32:04.631540 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:32:04.633771 (+ 2231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:32:04.633789 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:32:04.634616 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:32:04.634625 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:32:04.635010 (+ 385us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:32:04.635949 (+ 939us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:32:04.635951 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:32:04.635994 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:32:04.636080 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:32:04.636486 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:32:04.636500 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:32:04.636510 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 16:32:04.636707 (+ 197us) client_negotiation.cc:240] Negotiation successful
0926 16:32:04.637171 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":375,"threads_started":1}
I20260926 16:32:04.640057 15786 heartbeater.cc:347] Connected to a master server at 127.6.33.62:43141
I20260926 16:32:04.640405 15786 heartbeater.cc:464] Registering TS with master...
I20260926 16:32:04.641215 15786 heartbeater.cc:511] Master 127.6.33.62:43141 requested a full tablet report, sending...
I20260926 16:32:04.643558 15600 ts_manager.cc:194] Registered new tserver with Master: eeb85b519acf497fa536cda8be702b56 (127.6.33.1:46373)
I20260926 16:32:04.645875 15600 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.33.1:38527
W20260926 16:32:04.962877 15790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:32:04.963171 15790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:32:04.963263 15790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:32:04.970809 15790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:32:04.970920 15790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:32:04.971025 15790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.2
I20260926 16:32:04.980468 15790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:36619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.33.2
--webserver_port=0
--tserver_master_addrs=127.6.33.62:43141
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.2
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:32:04.981864 15790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:32:04.983645 15790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:32:04.997478 15795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:32:04.997439 15796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:32:04.997910 15798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:32:04.998714 15790 server_base.cc:1034] running on GCE node
I20260926 16:32:04.999313 15790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:32:05.000285 15790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:32:05.001507 15790 hybrid_clock.cc:648] HybridClock initialized: now 1790440325001451 us; error 46 us; skew 500 ppm
I20260926 16:32:05.001863 15790 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:32:05.004442 15790 webserver.cc:460] Webserver started at http://127.6.33.2:39933/ using document root <none> and password file <none>
I20260926 16:32:05.005395 15790 fs_manager.cc:359] Metadata directory not provided
I20260926 16:32:05.005528 15790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:32:05.005883 15790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:32:05.008984 15790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-1/data/instance:
uuid: "d9d7cf9f7f8e444c8756e7e74ef2d759"
format_stamp: "Formatted at 2026-09-26 16:32:05 on dist-test-slave-khd3"
server_key: "5890c469174a152da12569f4e7672bcb"
server_key_iv: "f57e34f1829fb69c197ab4a74094413f"
server_key_version: "encryptionkey@0"
I20260926 16:32:05.009869 15790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-1/wal/instance:
uuid: "d9d7cf9f7f8e444c8756e7e74ef2d759"
format_stamp: "Formatted at 2026-09-26 16:32:05 on dist-test-slave-khd3"
server_key: "5890c469174a152da12569f4e7672bcb"
server_key_iv: "f57e34f1829fb69c197ab4a74094413f"
server_key_version: "encryptionkey@0"
I20260926 16:32:05.015482 15790 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 16:32:05.019546 15804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:32:05.021196 15790 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 16:32:05.021385 15790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-1/wal
uuid: "d9d7cf9f7f8e444c8756e7e74ef2d759"
format_stamp: "Formatted at 2026-09-26 16:32:05 on dist-test-slave-khd3"
server_key: "5890c469174a152da12569f4e7672bcb"
server_key_iv: "f57e34f1829fb69c197ab4a74094413f"
server_key_version: "encryptionkey@0"
I20260926 16:32:05.021597 15790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:32:05.049137 15790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:32:05.050022 15790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:32:05.050326 15790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:32:05.051326 15790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:32:05.052911 15790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:32:05.053000 15790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:32:05.053195 15790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:32:05.053295 15790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:32:05.088825 15790 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.2:35043
I20260926 16:32:05.088898 15916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.2:35043 every 8 connection(s)
I20260926 16:32:05.090212 15790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
I20260926 16:32:05.096057 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 15790
I20260926 16:32:05.096249 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-1/wal/instance
I20260926 16:32:05.096670 6276 external_mini_cluster.cc:1428] Setting key 72baee433d603f078b0f43decd4d01e1
I20260926 16:32:05.099642 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.3:0
--local_ip_for_outbound_sockets=127.6.33.3
--webserver_interface=127.6.33.3
--webserver_port=0
--tserver_master_addrs=127.6.33.62:43141
--builtin_ntp_servers=127.6.33.20:36619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 16:32:05.103991 15788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:32:05.092847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:60277 (local address 127.6.33.62:43141)
0926 16:32:05.093045 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 16:32:05.093050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:32:05.094606 (+ 1556us) server_negotiation.cc:407] Connection header received
0926 16:32:05.096501 (+ 1895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:32:05.096506 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:32:05.096594 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:32:05.096702 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:32:05.098012 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:32:05.099039 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:32:05.100807 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:32:05.101545 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:32:05.101690 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:32:05.103070 (+ 1380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:32:05.103086 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:32:05.103094 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:32:05.103120 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:32:05.103164 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:32:05.103181 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:32:05.103274 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 16:32:05.103489 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 16:32:05.103598 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 16:32:05.105000 15919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:32:05.093446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:43141 (local address 127.6.33.2:60277)
0926 16:32:05.094458 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0926 16:32:05.094495 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 16:32:05.096180 (+ 1685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:32:05.096956 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:32:05.096964 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:32:05.097058 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:32:05.097810 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:32:05.097835 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:32:05.099237 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:32:05.099241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:32:05.100612 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:32:05.100625 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:32:05.101779 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:32:05.101784 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:32:05.101995 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:32:05.102749 (+ 754us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:32:05.102752 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:32:05.102792 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:32:05.102876 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:32:05.103283 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:32:05.103290 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:32:05.103296 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 16:32:05.103451 (+ 155us) client_negotiation.cc:240] Negotiation successful
0926 16:32:05.103779 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":330,"threads_started":1}
I20260926 16:32:05.106774 15917 heartbeater.cc:347] Connected to a master server at 127.6.33.62:43141
I20260926 16:32:05.107124 15917 heartbeater.cc:464] Registering TS with master...
I20260926 16:32:05.108194 15917 heartbeater.cc:511] Master 127.6.33.62:43141 requested a full tablet report, sending...
I20260926 16:32:05.110106 15600 ts_manager.cc:194] Registered new tserver with Master: d9d7cf9f7f8e444c8756e7e74ef2d759 (127.6.33.2:35043)
I20260926 16:32:05.111359 15600 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.33.2:60277
W20260926 16:32:05.445963 15920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:32:05.446276 15920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:32:05.446393 15920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:32:05.453887 15920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:32:05.454005 15920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:32:05.454110 15920 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.3
I20260926 16:32:05.463578 15920 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:36619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.33.3
--webserver_port=0
--tserver_master_addrs=127.6.33.62:43141
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.3
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:32:05.464854 15920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:32:05.466687 15920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:32:05.479525 15928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:32:05.479601 15925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:32:05.479887 15926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:32:05.480525 15920 server_base.cc:1034] running on GCE node
I20260926 16:32:05.481201 15920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:32:05.482084 15920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:32:05.483279 15920 hybrid_clock.cc:648] HybridClock initialized: now 1790440325483237 us; error 46 us; skew 500 ppm
I20260926 16:32:05.483608 15920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:32:05.486006 15920 webserver.cc:460] Webserver started at http://127.6.33.3:41677/ using document root <none> and password file <none>
I20260926 16:32:05.486745 15920 fs_manager.cc:359] Metadata directory not provided
I20260926 16:32:05.486830 15920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:32:05.487191 15920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:32:05.490253 15920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-2/data/instance:
uuid: "b20597869c62478abd7437378052c5b3"
format_stamp: "Formatted at 2026-09-26 16:32:05 on dist-test-slave-khd3"
server_key: "78ee5dcfd3592e1138a18b99b52f8822"
server_key_iv: "919e0f169b84f9d40419fb2aba4270a5"
server_key_version: "encryptionkey@0"
I20260926 16:32:05.491078 15920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-2/wal/instance:
uuid: "b20597869c62478abd7437378052c5b3"
format_stamp: "Formatted at 2026-09-26 16:32:05 on dist-test-slave-khd3"
server_key: "78ee5dcfd3592e1138a18b99b52f8822"
server_key_iv: "919e0f169b84f9d40419fb2aba4270a5"
server_key_version: "encryptionkey@0"
I20260926 16:32:05.496580 15920 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 16:32:05.500032 15934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:32:05.501657 15920 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:32:05.501850 15920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-2/wal
uuid: "b20597869c62478abd7437378052c5b3"
format_stamp: "Formatted at 2026-09-26 16:32:05 on dist-test-slave-khd3"
server_key: "78ee5dcfd3592e1138a18b99b52f8822"
server_key_iv: "919e0f169b84f9d40419fb2aba4270a5"
server_key_version: "encryptionkey@0"
I20260926 16:32:05.502081 15920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:32:05.521364 15920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:32:05.522212 15920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:32:05.522486 15920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:32:05.523434 15920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:32:05.524964 15920 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:32:05.525098 15920 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:32:05.525223 15920 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:32:05.525307 15920 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:32:05.559242 15920 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.3:32873
I20260926 16:32:05.559280 16046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.3:32873 every 8 connection(s)
I20260926 16:32:05.560912 15920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
I20260926 16:32:05.567121 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 15920
I20260926 16:32:05.567328 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-2/wal/instance
I20260926 16:32:05.567865 6276 external_mini_cluster.cc:1428] Setting key 52c477e5f973043b128ba1b39f05a208
I20260926 16:32:05.574725 15788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:32:05.563629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.3:60279 (local address 127.6.33.62:43141)
0926 16:32:05.563822 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 16:32:05.563826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:32:05.565363 (+ 1537us) server_negotiation.cc:407] Connection header received
0926 16:32:05.567362 (+ 1999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:32:05.567371 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:32:05.567472 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:32:05.567560 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:32:05.568991 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:32:05.570020 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:32:05.571842 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:32:05.572238 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:32:05.572339 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:32:05.574013 (+ 1674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:32:05.574018 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:32:05.574021 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:32:05.574039 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:32:05.574069 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:32:05.574081 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:32:05.574163 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 16:32:05.574406 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 16:32:05.574482 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 16:32:05.575681 16049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:32:05.564185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:43141 (local address 127.6.33.3:60279)
0926 16:32:05.565210 (+ 1025us) negotiation.cc:107] Waiting for socket to connect
0926 16:32:05.565252 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 16:32:05.567003 (+ 1751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:32:05.567852 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:32:05.567864 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:32:05.567991 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:32:05.568812 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:32:05.568836 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:32:05.570198 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:32:05.570202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:32:05.571690 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:32:05.571709 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:32:05.572482 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:32:05.572488 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:32:05.572934 (+ 446us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:32:05.573709 (+ 775us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:32:05.573712 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:32:05.573751 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:32:05.573831 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:32:05.574201 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:32:05.574209 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:32:05.574214 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 16:32:05.574350 (+ 136us) client_negotiation.cc:240] Negotiation successful
0926 16:32:05.574647 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":276,"threads_started":1}
I20260926 16:32:05.577045 16047 heartbeater.cc:347] Connected to a master server at 127.6.33.62:43141
I20260926 16:32:05.577342 16047 heartbeater.cc:464] Registering TS with master...
I20260926 16:32:05.578063 16047 heartbeater.cc:511] Master 127.6.33.62:43141 requested a full tablet report, sending...
I20260926 16:32:05.579473 15600 ts_manager.cc:194] Registered new tserver with Master: b20597869c62478abd7437378052c5b3 (127.6.33.3:32873)
I20260926 16:32:05.580250 15600 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.33.3:60279
I20260926 16:32:05.583892 6276 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 16:32:05.590162 15788 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:32:05.588597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47610 (local address 127.6.33.62:43141)
0926 16:32:05.588861 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0926 16:32:05.588868 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:32:05.589156 (+ 288us) server_negotiation.cc:407] Connection header received
0926 16:32:05.589333 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:32:05.589337 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:32:05.589404 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:32:05.589523 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:32:05.589934 (+ 411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47610: BlockingRecv error: recv got EOF from 127.0.0.1:47610 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 16:32:05.590842 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 15659
I20260926 16:32:05.599061 16056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:32:05.591840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35476 (local address 127.6.33.1:46373)
0926 16:32:05.592394 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0926 16:32:05.592405 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:32:05.592439 (+ 34us) server_negotiation.cc:407] Connection header received
0926 16:32:05.592518 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:32:05.592526 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:32:05.592872 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:32:05.593041 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:32:05.593453 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:32:05.594568 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:32:05.595879 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:32:05.596439 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:32:05.596544 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:32:05.597592 (+ 1048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:32:05.597618 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:32:05.597631 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:32:05.598218 (+ 587us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:32:05.598262 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:32:05.598280 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:32:05.598378 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 16:32:05.598521 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 16:32:05.598670 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":286,"threads_started":1}
I20260926 16:32:05.600310 15781 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:32:05.685022 15786 heartbeater.cc:503] Master 127.6.33.62:43141 was elected leader, sending a full tablet report...
I20260926 16:32:05.686264 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 15659
W20260926 16:32:05.702669 15584 connection.cc:535] server connection from 127.6.33.1:38527 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 16:32:05.702958 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 15790
I20260926 16:32:05.711174 16058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:32:05.703904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53604 (local address 127.6.33.2:35043)
0926 16:32:05.704470 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0926 16:32:05.704477 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:32:05.704502 (+ 25us) server_negotiation.cc:407] Connection header received
0926 16:32:05.704584 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:32:05.704590 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:32:05.704887 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:32:05.705063 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:32:05.705476 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:32:05.706634 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:32:05.707918 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:32:05.708491 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:32:05.708630 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:32:05.709751 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:32:05.709768 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:32:05.709781 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:32:05.710341 (+ 560us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:32:05.710385 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:32:05.710404 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:32:05.710538 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 16:32:05.710684 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 16:32:05.710827 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":322,"threads_started":1}
I20260926 16:32:05.712560 15912 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:32:05.795222 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 15790
I20260926 16:32:05.811699 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 15920
I20260926 16:32:05.820174 16060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:32:05.812889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43962 (local address 127.6.33.3:32873)
0926 16:32:05.813503 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0926 16:32:05.813510 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:32:05.813541 (+ 31us) server_negotiation.cc:407] Connection header received
0926 16:32:05.813601 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:32:05.813606 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:32:05.813858 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:32:05.813986 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:32:05.814416 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:32:05.815479 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:32:05.816960 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:32:05.817570 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:32:05.817692 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:32:05.818745 (+ 1053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:32:05.818753 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:32:05.818767 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:32:05.819361 (+ 594us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:32:05.819406 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:32:05.819425 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:32:05.819515 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 16:32:05.819642 (+ 127us) server_negotiation.cc:299] Negotiation successful
0926 16:32:05.819807 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":327,"threads_started":1}
I20260926 16:32:05.821550 16042 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:32:05.903263 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 15920
I20260926 16:32:05.919230 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 15569
I20260926 16:32:05.920457 15630 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:32:06.022337 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 15569
2026-09-26T16:32:06Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2473 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-26T16:32:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:32:06Z Disabled control of system clock
I20260926 16:32:06.095525 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440233850748-6276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440233850748-6276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440233850748-6276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:45109
--webserver_interface=127.6.33.62
--webserver_port=0
--builtin_ntp_servers=127.6.33.20:45531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.33.62:45109
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 16:32:06.427875 16066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:32:06.428211 16066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:32:06.428303 16066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:32:06.435832 16066 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:32:06.435997 16066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:32:06.436064 16066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:32:06.436157 16066 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:32:06.436206 16066 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:32:06.445925 16066 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:45531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440233850748-6276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440233850748-6276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.33.62:45109
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:45109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440233850748-6276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:32:06.447283 16066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:32:06.449087 16066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:32:06.461107 16074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:32:06.461237 16072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:32:06.461258 16071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:32:06.462054 16066 server_base.cc:1034] running on GCE node
I20260926 16:32:06.462935 16066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:32:06.464514 16066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:32:06.465782 16066 hybrid_clock.cc:648] HybridClock initialized: now 1790440326465714 us; error 67 us; skew 500 ppm
I20260926 16:32:06.466261 16066 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:32:06.468971 16066 webserver.cc:460] Webserver started at http://127.6.33.62:43505/ using document root <none> and password file <none>
I20260926 16:32:06.469750 16066 fs_manager.cc:359] Metadata directory not provided
I20260926 16:32:06.469892 16066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:32:06.470232 16066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:32:06.473474 16066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440233850748-6276-0/minicluster-data/master-0/data/instance:
uuid: "7737fb656fee4768928ab50d427d08b0"
format_stamp: "Formatted at 2026-09-26 16:32:06 on dist-test-slave-khd3"
server_key: "248fdae57285b74f6b59c63f0bd855b6"
server_key_iv: "a1ac05192dd55d297ddfcea982280ec6"
server_key_version: "encryptionkey@0"
I20260926 16:32:06.474345 16066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440233850748-6276-0/minicluster-data/master-0/wal/instance:
uuid: "7737fb656fee4768928ab50d427d08b0"
format_stamp: "Formatted at 2026-09-26 16:32:06 on dist-test-slave-khd3"
server_key: "248fdae57285b74f6b59c63f0bd855b6"
server_key_iv: "a1ac05192dd55d297ddfcea982280ec6"
server_key_version: "encryptionkey@0"
I20260926 16:32:06.479885 16066 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260926 16:32:06.483374 16080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:32:06.484894 16066 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 16:32:06.485069 16066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440233850748-6276-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440233850748-6276-0/minicluster-data/master-0/wal
uuid: "7737fb656fee4768928ab50d427d08b0"
format_stamp: "Formatted at 2026-09-26 16:32:06 on dist-test-slave-khd3"
server_key: "248fdae57285b74f6b59c63f0bd855b6"
server_key_iv: "a1ac05192dd55d297ddfcea982280ec6"
server_key_version: "encryptionkey@0"
I20260926 16:32:06.485287 16066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440233850748-6276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440233850748-6276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440233850748-6276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:32:06.491003 16066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:32:06.491776 16066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:32:06.492065 16066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:32:06.510778 16066 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.62:45109
I20260926 16:32:06.510836 16131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.62:45109 every 8 connection(s)
I20260926 16:32:06.512125 16066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
I20260926 16:32:06.515947 16132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:32:06.517915 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 16066
I20260926 16:32:06.518172 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440233850748-6276-0/minicluster-data/master-0/wal/instance
I20260926 16:32:06.518797 6276 external_mini_cluster.cc:1428] Setting key 0ea5f0cf58af9d654173ec1521f27f9c
I20260926 16:32:06.528640 16135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:32:06.521093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38824 (local address 127.6.33.62:45109)
0926 16:32:06.522195 (+ 1102us) server_negotiation.cc:206] Beginning negotiation
0926 16:32:06.522209 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 16:32:06.522258 (+ 49us) server_negotiation.cc:407] Connection header received
0926 16:32:06.524508 (+ 2250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:32:06.524539 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:32:06.525132 (+ 593us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:32:06.525582 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:32:06.526618 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:32:06.526626 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:32:06.526643 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:32:06.526660 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:32:06.526744 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:32:06.526765 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:32:06.526998 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0926 16:32:06.527149 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 16:32:06.527416 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":335,"threads_started":1}
I20260926 16:32:06.529314 16132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7737fb656fee4768928ab50d427d08b0: Bootstrap starting.
I20260926 16:32:06.532423 16132 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7737fb656fee4768928ab50d427d08b0: Neither blocks nor log segments found. Creating new log.
I20260926 16:32:06.533460 16132 log.cc:824] T 00000000000000000000000000000000 P 7737fb656fee4768928ab50d427d08b0: Log is configured to *not* fsync() on all Append() calls
I20260926 16:32:06.536185 16132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7737fb656fee4768928ab50d427d08b0: No bootstrap required, opened a new log
I20260926 16:32:06.544235 16132 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7737fb656fee4768928ab50d427d08b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7737fb656fee4768928ab50d427d08b0" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 45109 } }
I20260926 16:32:06.544808 16132 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7737fb656fee4768928ab50d427d08b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:32:06.544926 16132 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7737fb656fee4768928ab50d427d08b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7737fb656fee4768928ab50d427d08b0, State: Initialized, Role: FOLLOWER
I20260926 16:32:06.545550 16132 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7737fb656fee4768928ab50d427d08b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7737fb656fee4768928ab50d427d08b0" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 45109 } }
I20260926 16:32:06.545755 16132 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7737fb656fee4768928ab50d427d08b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:32:06.545912 16132 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7737fb656fee4768928ab50d427d08b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:32:06.546065 16132 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7737fb656fee4768928ab50d427d08b0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:32:06.548262 16132 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7737fb656fee4768928ab50d427d08b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7737fb656fee4768928ab50d427d08b0" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 45109 } }
I20260926 16:32:06.548837 16132 leader_election.cc:302] T 00000000000000000000000000000000 P 7737fb656fee4768928ab50d427d08b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7737fb656fee4768928ab50d427d08b0; no voters:
I20260926 16:32:06.549444 16132 leader_election.cc:288] T 00000000000000000000000000000000 P 7737fb656fee4768928ab50d427d08b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:32:06.549588 16137 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7737fb656fee4768928ab50d427d08b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:32:06.550351 16137 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7737fb656fee4768928ab50d427d08b0 [term 1 LEADER]: Becoming Leader. State: Replica: 7737fb656fee4768928ab50d427d08b0, State: Running, Role: LEADER
I20260926 16:32:06.550949 16137 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7737fb656fee4768928ab50d427d08b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7737fb656fee4768928ab50d427d08b0" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 45109 } }
I20260926 16:32:06.551614 16132 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7737fb656fee4768928ab50d427d08b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:32:06.553987 16138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7737fb656fee4768928ab50d427d08b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7737fb656fee4768928ab50d427d08b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7737fb656fee4768928ab50d427d08b0" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 45109 } } }
I20260926 16:32:06.554821 16139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7737fb656fee4768928ab50d427d08b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7737fb656fee4768928ab50d427d08b0. Latest consensus state: current_term: 1 leader_uuid: "7737fb656fee4768928ab50d427d08b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7737fb656fee4768928ab50d427d08b0" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 45109 } } }
I20260926 16:32:06.555349 16138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7737fb656fee4768928ab50d427d08b0 [sys.catalog]: This master's current role is: LEADER
I20260926 16:32:06.555441 16139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7737fb656fee4768928ab50d427d08b0 [sys.catalog]: This master's current role is: LEADER
I20260926 16:32:06.557906 16146 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:32:06.563468 16146 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:32:06.573063 16146 catalog_manager.cc:1347] Generated new cluster ID: c068c6fda13c4aa7b612385d2f5c5128
I20260926 16:32:06.573194 16146 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:32:06.595010 16146 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:32:06.595496 16146 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:32:06.604538 16146 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7737fb656fee4768928ab50d427d08b0: Generated new TSK 0
I20260926 16:32:06.605651 16146 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:32:06.617969 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.1:0
--local_ip_for_outbound_sockets=127.6.33.1
--webserver_interface=127.6.33.1
--webserver_port=0
--tserver_master_addrs=127.6.33.62:45109
--builtin_ntp_servers=127.6.33.20:45531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 16:32:06.955655 16156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:32:06.955992 16156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:32:06.956166 16156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:32:06.963828 16156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:32:06.963985 16156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:32:06.964180 16156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.1
I20260926 16:32:06.973604 16156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:45531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.33.1
--webserver_port=0
--tserver_master_addrs=127.6.33.62:45109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.1
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:32:06.974969 16156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:32:06.976846 16156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:32:06.991286 16161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:32:06.991375 16162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:32:06.991375 16164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:32:06.991628 16156 server_base.cc:1034] running on GCE node
I20260926 16:32:06.992687 16156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:32:06.993757 16156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:32:06.995009 16156 hybrid_clock.cc:648] HybridClock initialized: now 1790440326994967 us; error 41 us; skew 500 ppm
I20260926 16:32:06.995391 16156 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:32:06.997964 16156 webserver.cc:460] Webserver started at http://127.6.33.1:34601/ using document root <none> and password file <none>
I20260926 16:32:06.998745 16156 fs_manager.cc:359] Metadata directory not provided
I20260926 16:32:06.998883 16156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:32:06.999243 16156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:32:07.004057 16156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-0/data/instance:
uuid: "4d17e68abeff44a185e165d31b4e4dc3"
format_stamp: "Formatted at 2026-09-26 16:32:06 on dist-test-slave-khd3"
server_key: "65a4c620b856463809fa56cc114347e6"
server_key_iv: "0d31a55dc0ae0d12f96cc1bdb70f34e4"
server_key_version: "encryptionkey@0"
I20260926 16:32:07.005087 16156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-0/wal/instance:
uuid: "4d17e68abeff44a185e165d31b4e4dc3"
format_stamp: "Formatted at 2026-09-26 16:32:06 on dist-test-slave-khd3"
server_key: "65a4c620b856463809fa56cc114347e6"
server_key_iv: "0d31a55dc0ae0d12f96cc1bdb70f34e4"
server_key_version: "encryptionkey@0"
I20260926 16:32:07.011114 16156 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.001s
I20260926 16:32:07.015179 16170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:32:07.016865 16156 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:32:07.017060 16156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-0/wal
uuid: "4d17e68abeff44a185e165d31b4e4dc3"
format_stamp: "Formatted at 2026-09-26 16:32:06 on dist-test-slave-khd3"
server_key: "65a4c620b856463809fa56cc114347e6"
server_key_iv: "0d31a55dc0ae0d12f96cc1bdb70f34e4"
server_key_version: "encryptionkey@0"
I20260926 16:32:07.017305 16156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:32:07.023207 16156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:32:07.024060 16156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:32:07.024346 16156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:32:07.025365 16156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:32:07.026808 16156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:32:07.026925 16156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:32:07.027062 16156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:32:07.027150 16156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:32:07.062400 16156 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.1:42451
I20260926 16:32:07.062455 16282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.1:42451 every 8 connection(s)
I20260926 16:32:07.063802 16156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
I20260926 16:32:07.072430 16285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:32:07.066654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:50469 (local address 127.6.33.62:45109)
0926 16:32:07.067142 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0926 16:32:07.067147 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:32:07.068428 (+ 1281us) server_negotiation.cc:407] Connection header received
0926 16:32:07.070275 (+ 1847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:32:07.070280 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:32:07.070334 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:32:07.070448 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:32:07.071820 (+ 1372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:32:07.071824 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:32:07.071827 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:32:07.071842 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:32:07.071892 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:32:07.071903 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:32:07.072001 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 16:32:07.072162 (+ 161us) server_negotiation.cc:299] Negotiation successful
0926 16:32:07.072212 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":223,"threads_started":1}
I20260926 16:32:07.073477 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 16156
I20260926 16:32:07.073501 16286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:32:07.067304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:45109 (local address 127.6.33.1:50469)
0926 16:32:07.068271 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0926 16:32:07.068315 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 16:32:07.069991 (+ 1676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:32:07.070669 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:32:07.070677 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:32:07.070774 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:32:07.071525 (+ 751us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:32:07.071528 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:32:07.071567 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:32:07.071656 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:32:07.071989 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:32:07.071995 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:32:07.072000 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 16:32:07.072250 (+ 250us) client_negotiation.cc:240] Negotiation successful
0926 16:32:07.072450 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":300,"threads_started":1}
I20260926 16:32:07.073660 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-0/wal/instance
I20260926 16:32:07.074432 6276 external_mini_cluster.cc:1428] Setting key 4f8eec0a927c6c1223d07ce63b696dcc
I20260926 16:32:07.075204 16283 heartbeater.cc:347] Connected to a master server at 127.6.33.62:45109
I20260926 16:32:07.075533 16283 heartbeater.cc:464] Registering TS with master...
I20260926 16:32:07.076241 16283 heartbeater.cc:511] Master 127.6.33.62:45109 requested a full tablet report, sending...
I20260926 16:32:07.077641 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.2:0
--local_ip_for_outbound_sockets=127.6.33.2
--webserver_interface=127.6.33.2
--webserver_port=0
--tserver_master_addrs=127.6.33.62:45109
--builtin_ntp_servers=127.6.33.20:45531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 16:32:07.078722 16097 ts_manager.cc:194] Registered new tserver with Master: 4d17e68abeff44a185e165d31b4e4dc3 (127.6.33.1:42451)
W20260926 16:32:07.405612 16287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:32:07.405893 16287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:32:07.405979 16287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:32:07.413128 16287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:32:07.413223 16287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:32:07.413329 16287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.2
I20260926 16:32:07.422356 16287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:45531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.33.2
--webserver_port=0
--tserver_master_addrs=127.6.33.62:45109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.2
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:32:07.423581 16287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:32:07.425284 16287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:32:07.438403 16292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:32:07.438336 16295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:32:07.438336 16293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:32:07.439211 16287 server_base.cc:1034] running on GCE node
I20260926 16:32:07.439815 16287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:32:07.440852 16287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:32:07.442131 16287 hybrid_clock.cc:648] HybridClock initialized: now 1790440327442073 us; error 46 us; skew 500 ppm
I20260926 16:32:07.442566 16287 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:32:07.445405 16287 webserver.cc:460] Webserver started at http://127.6.33.2:43047/ using document root <none> and password file <none>
I20260926 16:32:07.446179 16287 fs_manager.cc:359] Metadata directory not provided
I20260926 16:32:07.446337 16287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:32:07.446702 16287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:32:07.449965 16287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-1/data/instance:
uuid: "9f2909b02a9f4de2be90e7fc3e4ebf56"
format_stamp: "Formatted at 2026-09-26 16:32:07 on dist-test-slave-khd3"
server_key: "6fa4beacc1ebf5e1e580b57659acfecb"
server_key_iv: "d170556922f4e38a8872bb3b98291be1"
server_key_version: "encryptionkey@0"
I20260926 16:32:07.450856 16287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-1/wal/instance:
uuid: "9f2909b02a9f4de2be90e7fc3e4ebf56"
format_stamp: "Formatted at 2026-09-26 16:32:07 on dist-test-slave-khd3"
server_key: "6fa4beacc1ebf5e1e580b57659acfecb"
server_key_iv: "d170556922f4e38a8872bb3b98291be1"
server_key_version: "encryptionkey@0"
I20260926 16:32:07.456722 16287 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 16:32:07.460616 16301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:32:07.462777 16287 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 16:32:07.462961 16287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-1/wal
uuid: "9f2909b02a9f4de2be90e7fc3e4ebf56"
format_stamp: "Formatted at 2026-09-26 16:32:07 on dist-test-slave-khd3"
server_key: "6fa4beacc1ebf5e1e580b57659acfecb"
server_key_iv: "d170556922f4e38a8872bb3b98291be1"
server_key_version: "encryptionkey@0"
I20260926 16:32:07.463243 16287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:32:07.469511 16287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:32:07.470343 16287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:32:07.470634 16287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:32:07.471590 16287 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:32:07.473112 16287 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:32:07.473237 16287 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:32:07.473368 16287 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:32:07.473454 16287 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:32:07.507359 16287 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.2:33091
I20260926 16:32:07.507411 16413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.2:33091 every 8 connection(s)
I20260926 16:32:07.508694 16287 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
I20260926 16:32:07.509902 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 16287
I20260926 16:32:07.510082 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-1/wal/instance
I20260926 16:32:07.510607 6276 external_mini_cluster.cc:1428] Setting key 458e9486ebc1dfcbcfaa9f5c7386d4e1
I20260926 16:32:07.513739 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.3:0
--local_ip_for_outbound_sockets=127.6.33.3
--webserver_interface=127.6.33.3
--webserver_port=0
--tserver_master_addrs=127.6.33.62:45109
--builtin_ntp_servers=127.6.33.20:45531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 16:32:07.518915 16285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:32:07.511657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:44025 (local address 127.6.33.62:45109)
0926 16:32:07.511855 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 16:32:07.511860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:32:07.513597 (+ 1737us) server_negotiation.cc:407] Connection header received
0926 16:32:07.516333 (+ 2736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:32:07.516339 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:32:07.516417 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:32:07.516523 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:32:07.518152 (+ 1629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:32:07.518160 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:32:07.518167 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:32:07.518192 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:32:07.518229 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:32:07.518244 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:32:07.518361 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 16:32:07.518571 (+ 210us) server_negotiation.cc:299] Negotiation successful
0926 16:32:07.518636 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 16:32:07.520202 16416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:32:07.512254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:45109 (local address 127.6.33.2:44025)
0926 16:32:07.513418 (+ 1164us) negotiation.cc:107] Waiting for socket to connect
0926 16:32:07.513465 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 16:32:07.516015 (+ 2550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:32:07.516815 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:32:07.516828 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:32:07.516958 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:32:07.517764 (+ 806us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:32:07.517770 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:32:07.517822 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:32:07.517938 (+ 116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:32:07.518352 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:32:07.518364 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:32:07.518375 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 16:32:07.518644 (+ 269us) client_negotiation.cc:240] Negotiation successful
0926 16:32:07.518913 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":343,"threads_started":1}
I20260926 16:32:07.521940 16414 heartbeater.cc:347] Connected to a master server at 127.6.33.62:45109
I20260926 16:32:07.522292 16414 heartbeater.cc:464] Registering TS with master...
I20260926 16:32:07.523026 16414 heartbeater.cc:511] Master 127.6.33.62:45109 requested a full tablet report, sending...
I20260926 16:32:07.524516 16097 ts_manager.cc:194] Registered new tserver with Master: 9f2909b02a9f4de2be90e7fc3e4ebf56 (127.6.33.2:33091)
W20260926 16:32:07.849093 16417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:32:07.849408 16417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:32:07.849566 16417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:32:07.856904 16417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:32:07.857038 16417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:32:07.857255 16417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.3
I20260926 16:32:07.866746 16417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:45531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.33.3
--webserver_port=0
--tserver_master_addrs=127.6.33.62:45109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.3
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:32:07.868119 16417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:32:07.869931 16417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:32:07.882951 16422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:32:07.882859 16425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:32:07.882926 16423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:32:07.883750 16417 server_base.cc:1034] running on GCE node
I20260926 16:32:07.884523 16417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:32:07.885577 16417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:32:07.886807 16417 hybrid_clock.cc:648] HybridClock initialized: now 1790440327886765 us; error 42 us; skew 500 ppm
I20260926 16:32:07.887142 16417 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:32:07.889655 16417 webserver.cc:460] Webserver started at http://127.6.33.3:45289/ using document root <none> and password file <none>
I20260926 16:32:07.890372 16417 fs_manager.cc:359] Metadata directory not provided
I20260926 16:32:07.890504 16417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:32:07.890834 16417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:32:07.893761 16417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-2/data/instance:
uuid: "5a7df408319343aebd23b039330e52a2"
format_stamp: "Formatted at 2026-09-26 16:32:07 on dist-test-slave-khd3"
server_key: "1f710116efbb0b1564790c7dd1d9e6e6"
server_key_iv: "d574e6f8635214b36eb8cb63f2746985"
server_key_version: "encryptionkey@0"
I20260926 16:32:07.894565 16417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-2/wal/instance:
uuid: "5a7df408319343aebd23b039330e52a2"
format_stamp: "Formatted at 2026-09-26 16:32:07 on dist-test-slave-khd3"
server_key: "1f710116efbb0b1564790c7dd1d9e6e6"
server_key_iv: "d574e6f8635214b36eb8cb63f2746985"
server_key_version: "encryptionkey@0"
I20260926 16:32:07.899899 16417 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260926 16:32:07.903677 16431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:32:07.905342 16417 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:32:07.905530 16417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-2/wal
uuid: "5a7df408319343aebd23b039330e52a2"
format_stamp: "Formatted at 2026-09-26 16:32:07 on dist-test-slave-khd3"
server_key: "1f710116efbb0b1564790c7dd1d9e6e6"
server_key_iv: "d574e6f8635214b36eb8cb63f2746985"
server_key_version: "encryptionkey@0"
I20260926 16:32:07.905759 16417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:32:07.911857 16417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:32:07.912717 16417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:32:07.913110 16417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:32:07.914024 16417 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:32:07.915482 16417 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:32:07.915604 16417 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:32:07.915745 16417 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:32:07.915833 16417 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:32:07.950717 16417 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.3:32845
I20260926 16:32:07.950817 16543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.3:32845 every 8 connection(s)
I20260926 16:32:07.952148 16417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
I20260926 16:32:07.960427 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 16417
I20260926 16:32:07.960608 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440233850748-6276-0/minicluster-data/ts-2/wal/instance
I20260926 16:32:07.960753 16285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:32:07.954849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.3:58439 (local address 127.6.33.62:45109)
0926 16:32:07.955021 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 16:32:07.955026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:32:07.956455 (+ 1429us) server_negotiation.cc:407] Connection header received
0926 16:32:07.958267 (+ 1812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:32:07.958272 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:32:07.958339 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:32:07.958448 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:32:07.959834 (+ 1386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:32:07.959840 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:32:07.959846 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:32:07.959867 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:32:07.959910 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:32:07.959930 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:32:07.960081 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 16:32:07.960363 (+ 282us) server_negotiation.cc:299] Negotiation successful
0926 16:32:07.960417 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 16:32:07.961217 6276 external_mini_cluster.cc:1428] Setting key 355b2b3cc591213f4e532657fbf3cccc
I20260926 16:32:07.961979 16546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:32:07.955337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:45109 (local address 127.6.33.3:58439)
0926 16:32:07.956289 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0926 16:32:07.956334 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 16:32:07.957968 (+ 1634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:32:07.958727 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:32:07.958737 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:32:07.958834 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:32:07.959484 (+ 650us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:32:07.959486 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:32:07.959524 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:32:07.959608 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:32:07.960058 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:32:07.960068 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:32:07.960077 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 16:32:07.960451 (+ 374us) client_negotiation.cc:240] Negotiation successful
0926 16:32:07.960740 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":305,"threads_started":1}
I20260926 16:32:07.963474 16544 heartbeater.cc:347] Connected to a master server at 127.6.33.62:45109
I20260926 16:32:07.963778 16544 heartbeater.cc:464] Registering TS with master...
I20260926 16:32:07.964444 16544 heartbeater.cc:511] Master 127.6.33.62:45109 requested a full tablet report, sending...
I20260926 16:32:07.965867 16097 ts_manager.cc:194] Registered new tserver with Master: 5a7df408319343aebd23b039330e52a2 (127.6.33.3:32845)
I20260926 16:32:07.977236 6276 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 16:32:07.983911 16285 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:32:07.982095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38828 (local address 127.6.33.62:45109)
0926 16:32:07.982322 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 16:32:07.982331 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:32:07.982888 (+ 557us) server_negotiation.cc:407] Connection header received
0926 16:32:07.983038 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:32:07.983042 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:32:07.983105 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:32:07.983187 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:32:07.983684 (+ 497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38828: BlockingRecv error: recv got EOF from 127.0.0.1:38828 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
I20260926 16:32:07.984614 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 16156
I20260926 16:32:07.988644 16553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:32:07.985529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38344 (local address 127.6.33.1:42451)
0926 16:32:07.986247 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0926 16:32:07.986258 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:32:07.986451 (+ 193us) server_negotiation.cc:407] Connection header received
0926 16:32:07.986525 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:32:07.986531 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:32:07.986756 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:32:07.986894 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:32:07.987836 (+ 942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:32:07.987843 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:32:07.987855 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:32:07.987873 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:32:07.987914 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:32:07.987929 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:32:07.988036 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 16:32:07.988248 (+ 212us) server_negotiation.cc:299] Negotiation successful
0926 16:32:07.988389 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":360,"threads_started":1}
I20260926 16:32:07.989856 16278 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:32:08.073900 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 16156
I20260926 16:32:08.089633 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 16287
I20260926 16:32:08.093292 16555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:32:08.090598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43040 (local address 127.6.33.2:33091)
0926 16:32:08.091140 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0926 16:32:08.091148 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:32:08.091215 (+ 67us) server_negotiation.cc:407] Connection header received
0926 16:32:08.091278 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:32:08.091284 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:32:08.091508 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:32:08.091634 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:32:08.092573 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:32:08.092580 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:32:08.092592 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:32:08.092604 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:32:08.092651 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:32:08.092671 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:32:08.092842 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 16:32:08.092938 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 16:32:08.093024 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":272,"threads_started":1}
I20260926 16:32:08.094458 16409 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:32:08.176704 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 16287
I20260926 16:32:08.192871 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 16417
I20260926 16:32:08.196965 16557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:32:08.193946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46000 (local address 127.6.33.3:32845)
0926 16:32:08.194657 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0926 16:32:08.194664 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:32:08.194695 (+ 31us) server_negotiation.cc:407] Connection header received
0926 16:32:08.194761 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:32:08.194767 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:32:08.195004 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:32:08.195182 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:32:08.196094 (+ 912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:32:08.196100 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:32:08.196111 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:32:08.196138 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:32:08.196177 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:32:08.196190 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:32:08.196321 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 16:32:08.196477 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 16:32:08.196556 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":397,"threads_started":1}
I20260926 16:32:08.198294 16539 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:32:08.281885 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 16417
I20260926 16:32:08.297837 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 16066
I20260926 16:32:08.299017 16127 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:32:08.400446 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 16066
2026-09-26T16:32:08Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2378 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:32:08 dist-test-slave-khd3 krb5kdc[16562](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:32:08 dist-test-slave-khd3 krb5kdc[16562](info): set up 2 sockets
Sep 26 16:32:08 dist-test-slave-khd3 krb5kdc[16562](info): commencing operation
krb5kdc: starting...
W20260926 16:32:09.507701 6276 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Sep 26 16:32:09 dist-test-slave-khd3 krb5kdc[16562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:32:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:32:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab.
Entry for principal kudu/127.6.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab.
I20260926 16:32:09.752743 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:44885
--webserver_interface=127.6.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:33815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.33.62:44885
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:32:10.090045 16575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:32:10.090364 16575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:32:10.090528 16575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:32:10.098376 16575 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:32:10.098495 16575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:32:10.098618 16575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:32:10.098722 16575 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:32:10.098810 16575 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:32:10.108837 16575 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:33815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.33.62:44885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:44885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:32:10.110217 16575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:32:10.111557 16575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:32:10.123358 16583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:32:10.123425 16581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:32:10.123329 16580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:32:10.124538 16575 server_base.cc:1034] running on GCE node
I20260926 16:32:10.125411 16575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:32:10.126866 16575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:32:10.128082 16575 hybrid_clock.cc:648] HybridClock initialized: now 1790440330128036 us; error 42 us; skew 500 ppm
I20260926 16:32:10.128450 16575 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:32:10 dist-test-slave-khd3 krb5kdc[16562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440330, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:32:10.140062 16575 init.cc:377] Logged in from keytab as kudu/127.6.33.62@KRBTEST.COM (short username kudu)
I20260926 16:32:10.141883 16575 webserver.cc:460] Webserver started at http://127.6.33.62:39969/ using document root <none> and password file <none>
I20260926 16:32:10.142668 16575 fs_manager.cc:359] Metadata directory not provided
I20260926 16:32:10.142818 16575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:32:10.143173 16575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:32:10.146516 16575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/master-0/data/instance:
uuid: "c64454921d8049efa0c88749678424be"
format_stamp: "Formatted at 2026-09-26 16:32:10 on dist-test-slave-khd3"
server_key: "57ad02639826f275336657d1a791a886"
server_key_iv: "5d48641e3fef951488f59235742eb641"
server_key_version: "encryptionkey@0"
I20260926 16:32:10.147411 16575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/master-0/wal/instance:
uuid: "c64454921d8049efa0c88749678424be"
format_stamp: "Formatted at 2026-09-26 16:32:10 on dist-test-slave-khd3"
server_key: "57ad02639826f275336657d1a791a886"
server_key_iv: "5d48641e3fef951488f59235742eb641"
server_key_version: "encryptionkey@0"
I20260926 16:32:10.153174 16575 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260926 16:32:10.157332 16590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:32:10.158962 16575 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 16:32:10.159181 16575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/master-0/wal
uuid: "c64454921d8049efa0c88749678424be"
format_stamp: "Formatted at 2026-09-26 16:32:10 on dist-test-slave-khd3"
server_key: "57ad02639826f275336657d1a791a886"
server_key_iv: "5d48641e3fef951488f59235742eb641"
server_key_version: "encryptionkey@0"
I20260926 16:32:10.159391 16575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:32:10.180485 16575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:32:10.185107 16575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:32:10.185432 16575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:32:10.205291 16575 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.62:44885
I20260926 16:32:10.205313 16641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.62:44885 every 8 connection(s)
I20260926 16:32:10.206768 16575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
I20260926 16:32:10.210884 16642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:32:10.216192 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 16575
I20260926 16:32:10.216420 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/master-0/wal/instance
I20260926 16:32:10.217276 6276 external_mini_cluster.cc:1428] Setting key 7d872849b20cd85f194c7dfb8dbb82ac
I20260926 16:32:10.223121 16642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c64454921d8049efa0c88749678424be: Bootstrap starting.
I20260926 16:32:10.226897 16642 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c64454921d8049efa0c88749678424be: Neither blocks nor log segments found. Creating new log.
I20260926 16:32:10.228201 16642 log.cc:824] T 00000000000000000000000000000000 P c64454921d8049efa0c88749678424be: Log is configured to *not* fsync() on all Append() calls
I20260926 16:32:10.231925 16642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c64454921d8049efa0c88749678424be: No bootstrap required, opened a new log
Sep 26 16:32:10 dist-test-slave-khd3 krb5kdc[16562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:32:10.240118 16642 raft_consensus.cc:348] T 00000000000000000000000000000000 P c64454921d8049efa0c88749678424be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c64454921d8049efa0c88749678424be" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 44885 } }
I20260926 16:32:10.240561 16642 raft_consensus.cc:374] T 00000000000000000000000000000000 P c64454921d8049efa0c88749678424be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:32:10.240689 16642 raft_consensus.cc:729] T 00000000000000000000000000000000 P c64454921d8049efa0c88749678424be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64454921d8049efa0c88749678424be, State: Initialized, Role: FOLLOWER
I20260926 16:32:10.241407 16642 consensus_queue.cc:260] T 00000000000000000000000000000000 P c64454921d8049efa0c88749678424be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c64454921d8049efa0c88749678424be" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 44885 } }
I20260926 16:32:10.241663 16642 raft_consensus.cc:388] T 00000000000000000000000000000000 P c64454921d8049efa0c88749678424be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:32:10.241802 16642 raft_consensus.cc:482] T 00000000000000000000000000000000 P c64454921d8049efa0c88749678424be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:32:10.242000 16642 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c64454921d8049efa0c88749678424be [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:32:10.242357 16645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:32:10.219544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48684 (local address 127.6.33.62:44885)
0926 16:32:10.220440 (+ 896us) server_negotiation.cc:206] Beginning negotiation
0926 16:32:10.220454 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 16:32:10.220504 (+ 50us) server_negotiation.cc:407] Connection header received
0926 16:32:10.222232 (+ 1728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:32:10.222257 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:32:10.222668 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:32:10.223016 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:32:10.224927 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:32:10.226302 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:32:10.228173 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:32:10.229216 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:32:10.229422 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:32:10.234123 (+ 4701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:32:10.234131 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:32:10.234148 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:32:10.234204 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:32:10.237668 (+ 3464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:32:10.238482 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:32:10.238490 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:32:10.238509 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:32:10.238698 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:32:10.239125 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:32:10.239130 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:32:10.239137 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:32:10.239715 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:32:10.240030 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0926 16:32:10.240543 (+ 513us) server_negotiation.cc:299] Negotiation successful
0926 16:32:10.241023 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":430,"threads_started":1}
I20260926 16:32:10.244518 16642 raft_consensus.cc:504] T 00000000000000000000000000000000 P c64454921d8049efa0c88749678424be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c64454921d8049efa0c88749678424be" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 44885 } }
I20260926 16:32:10.245105 16642 leader_election.cc:302] T 00000000000000000000000000000000 P c64454921d8049efa0c88749678424be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c64454921d8049efa0c88749678424be; no voters:
I20260926 16:32:10.245803 16642 leader_election.cc:288] T 00000000000000000000000000000000 P c64454921d8049efa0c88749678424be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:32:10.245925 16647 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c64454921d8049efa0c88749678424be [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:32:10.246627 16647 raft_consensus.cc:686] T 00000000000000000000000000000000 P c64454921d8049efa0c88749678424be [term 1 LEADER]: Becoming Leader. State: Replica: c64454921d8049efa0c88749678424be, State: Running, Role: LEADER
I20260926 16:32:10.247288 16647 consensus_queue.cc:237] T 00000000000000000000000000000000 P c64454921d8049efa0c88749678424be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c64454921d8049efa0c88749678424be" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 44885 } }
I20260926 16:32:10.248135 16642 sys_catalog.cc:564] T 00000000000000000000000000000000 P c64454921d8049efa0c88749678424be [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:32:10.250204 16648 sys_catalog.cc:455] T 00000000000000000000000000000000 P c64454921d8049efa0c88749678424be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c64454921d8049efa0c88749678424be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c64454921d8049efa0c88749678424be" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 44885 } } }
I20260926 16:32:10.250591 16648 sys_catalog.cc:458] T 00000000000000000000000000000000 P c64454921d8049efa0c88749678424be [sys.catalog]: This master's current role is: LEADER
I20260926 16:32:10.251335 16649 sys_catalog.cc:455] T 00000000000000000000000000000000 P c64454921d8049efa0c88749678424be [sys.catalog]: SysCatalogTable state changed. Reason: New leader c64454921d8049efa0c88749678424be. Latest consensus state: current_term: 1 leader_uuid: "c64454921d8049efa0c88749678424be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c64454921d8049efa0c88749678424be" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 44885 } } }
I20260926 16:32:10.251574 16654 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:32:10.251729 16649 sys_catalog.cc:458] T 00000000000000000000000000000000 P c64454921d8049efa0c88749678424be [sys.catalog]: This master's current role is: LEADER
I20260926 16:32:10.258335 16654 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:32:10.269376 16654 catalog_manager.cc:1347] Generated new cluster ID: dbcba91d532f427484305d1537eb35a7
I20260926 16:32:10.269516 16654 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:32:10.282281 16654 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:32:10.283967 16654 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:32:10.302875 16654 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c64454921d8049efa0c88749678424be: Generated new TSK 0
I20260926 16:32:10.303992 16654 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.33.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab.
Entry for principal kudu/127.6.33.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab.
I20260926 16:32:10.381884 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.1:0
--local_ip_for_outbound_sockets=127.6.33.1
--webserver_interface=127.6.33.1
--webserver_port=0
--tserver_master_addrs=127.6.33.62:44885
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab
--principal=kudu/127.6.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:33815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:32:10.733259 16668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:32:10.733560 16668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:32:10.733712 16668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:32:10.741626 16668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:32:10.741793 16668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:32:10.742026 16668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.1
I20260926 16:32:10.752291 16668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:33815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab
--principal=kudu/127.6.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.33.1
--webserver_port=0
--tserver_master_addrs=127.6.33.62:44885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.1
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:32:10.753595 16668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:32:10.754897 16668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:32:10.769843 16673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:32:10.769771 16674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:32:10.769868 16676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:32:10.770888 16668 server_base.cc:1034] running on GCE node
I20260926 16:32:10.771538 16668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:32:10.772527 16668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:32:10.773759 16668 hybrid_clock.cc:648] HybridClock initialized: now 1790440330773688 us; error 72 us; skew 500 ppm
I20260926 16:32:10.774089 16668 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:32:10 dist-test-slave-khd3 krb5kdc[16562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440330, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:32:10.777216 16668 init.cc:377] Logged in from keytab as kudu/127.6.33.1@KRBTEST.COM (short username kudu)
I20260926 16:32:10.778834 16668 webserver.cc:460] Webserver started at http://127.6.33.1:40883/ using document root <none> and password file <none>
I20260926 16:32:10.779482 16668 fs_manager.cc:359] Metadata directory not provided
I20260926 16:32:10.779580 16668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:32:10.779860 16668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:32:10.782801 16668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/ts-0/data/instance:
uuid: "944e1cbe75934ca0b6239d3e98ccd5c4"
format_stamp: "Formatted at 2026-09-26 16:32:10 on dist-test-slave-khd3"
server_key: "f6f9b7641b27557fb7b94cdab663a1ce"
server_key_iv: "41e9dd7fc347fd250a80f56773bc58d9"
server_key_version: "encryptionkey@0"
I20260926 16:32:10.783526 16668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/ts-0/wal/instance:
uuid: "944e1cbe75934ca0b6239d3e98ccd5c4"
format_stamp: "Formatted at 2026-09-26 16:32:10 on dist-test-slave-khd3"
server_key: "f6f9b7641b27557fb7b94cdab663a1ce"
server_key_iv: "41e9dd7fc347fd250a80f56773bc58d9"
server_key_version: "encryptionkey@0"
I20260926 16:32:10.788844 16668 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260926 16:32:10.792356 16683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:32:10.793772 16668 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 16:32:10.793910 16668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/ts-0/wal
uuid: "944e1cbe75934ca0b6239d3e98ccd5c4"
format_stamp: "Formatted at 2026-09-26 16:32:10 on dist-test-slave-khd3"
server_key: "f6f9b7641b27557fb7b94cdab663a1ce"
server_key_iv: "41e9dd7fc347fd250a80f56773bc58d9"
server_key_version: "encryptionkey@0"
I20260926 16:32:10.794073 16668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:32:10.806412 16668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:32:10.810853 16668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:32:10.811094 16668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:32:10.811990 16668 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:32:10.813344 16668 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:32:10.813416 16668 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:32:10.813498 16668 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:32:10.813540 16668 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:32:10.847849 16668 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.1:43869
I20260926 16:32:10.847990 16795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.1:43869 every 8 connection(s)
I20260926 16:32:10.849314 16668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
I20260926 16:32:10.850096 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 16668
I20260926 16:32:10.850297 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/ts-0/wal/instance
I20260926 16:32:10.851120 6276 external_mini_cluster.cc:1428] Setting key dcd39d4e310d7f559d9366f09c498be4
Sep 26 16:32:10 dist-test-slave-khd3 krb5kdc[16562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440330, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.1@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:32:10.873203 16798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:32:10.852351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:33613 (local address 127.6.33.62:44885)
0926 16:32:10.852857 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0926 16:32:10.852862 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:32:10.853988 (+ 1126us) server_negotiation.cc:407] Connection header received
0926 16:32:10.855890 (+ 1902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:32:10.855895 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:32:10.855982 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:32:10.856152 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:32:10.857782 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:32:10.858548 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:32:10.860591 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:32:10.861067 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:32:10.861186 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:32:10.866150 (+ 4964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:32:10.866156 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:32:10.866160 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:32:10.866196 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:32:10.869006 (+ 2810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:32:10.870046 (+ 1040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:32:10.870050 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:32:10.870052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:32:10.870183 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:32:10.870867 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:32:10.870871 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:32:10.870874 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:32:10.871267 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:32:10.871412 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 16:32:10.872544 (+ 1132us) server_negotiation.cc:299] Negotiation successful
0926 16:32:10.872744 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":289,"threads_started":1}
I20260926 16:32:10.874675 16799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:32:10.852992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:44885 (local address 127.6.33.1:33613)
0926 16:32:10.853839 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0926 16:32:10.853876 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 16:32:10.855599 (+ 1723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:32:10.856390 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:32:10.856399 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:32:10.857033 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:32:10.857616 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:32:10.857634 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:32:10.858720 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:32:10.858724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:32:10.860402 (+ 1678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:32:10.860420 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:32:10.861302 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:32:10.861312 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:32:10.861653 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:32:10.862526 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:32:10.862561 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:32:10.865873 (+ 3312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:32:10.869172 (+ 3299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:32:10.869186 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:32:10.869208 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:32:10.869892 (+ 684us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:32:10.870309 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:32:10.870317 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:32:10.870323 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:32:10.870585 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:32:10.871426 (+ 841us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:32:10.871439 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:32:10.871997 (+ 558us) client_negotiation.cc:769] Sending connection context
0926 16:32:10.872404 (+ 407us) client_negotiation.cc:240] Negotiation successful
0926 16:32:10.873108 (+ 704us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":382,"threads_started":1}
WARNING: no policy specified for kudu/127.6.33.2@KRBTEST.COM; defaulting to no policy
I20260926 16:32:10.876426 16796 heartbeater.cc:347] Connected to a master server at 127.6.33.62:44885
I20260926 16:32:10.876871 16796 heartbeater.cc:464] Registering TS with master...
I20260926 16:32:10.877652 16796 heartbeater.cc:511] Master 127.6.33.62:44885 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.2@KRBTEST.COM" created.
I20260926 16:32:10.880128 16607 ts_manager.cc:194] Registered new tserver with Master: 944e1cbe75934ca0b6239d3e98ccd5c4 (127.6.33.1:43869)
I20260926 16:32:10.881832 16607 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.33.1@KRBTEST.COM'} at 127.6.33.1:33613
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab.
Entry for principal kudu/127.6.33.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab.
I20260926 16:32:10.910331 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.2:0
--local_ip_for_outbound_sockets=127.6.33.2
--webserver_interface=127.6.33.2
--webserver_port=0
--tserver_master_addrs=127.6.33.62:44885
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab
--principal=kudu/127.6.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:33815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:32:11.253028 16802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:32:11.253329 16802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:32:11.253432 16802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:32:11.260915 16802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:32:11.261029 16802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:32:11.261216 16802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.2
I20260926 16:32:11.271152 16802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:33815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab
--principal=kudu/127.6.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.33.2
--webserver_port=0
--tserver_master_addrs=127.6.33.62:44885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.2
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:32:11.272433 16802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:32:11.273810 16802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:32:11.287508 16807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:32:11.287432 16808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:32:11.287470 16810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:32:11.288337 16802 server_base.cc:1034] running on GCE node
I20260926 16:32:11.289058 16802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:32:11.290000 16802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:32:11.291267 16802 hybrid_clock.cc:648] HybridClock initialized: now 1790440331291215 us; error 56 us; skew 500 ppm
I20260926 16:32:11.291669 16802 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:32:11 dist-test-slave-khd3 krb5kdc[16562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440331, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:32:11.294914 16802 init.cc:377] Logged in from keytab as kudu/127.6.33.2@KRBTEST.COM (short username kudu)
I20260926 16:32:11.296437 16802 webserver.cc:460] Webserver started at http://127.6.33.2:33093/ using document root <none> and password file <none>
I20260926 16:32:11.297331 16802 fs_manager.cc:359] Metadata directory not provided
I20260926 16:32:11.297472 16802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:32:11.297832 16802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:32:11.300810 16802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/ts-1/data/instance:
uuid: "bd748cfeb19d4dd8ac3a8c3407a19b18"
format_stamp: "Formatted at 2026-09-26 16:32:11 on dist-test-slave-khd3"
server_key: "b2bd0f2c15da3c0de82405808ecf2473"
server_key_iv: "5aacc633c793dc2bb7ccba7a874f55f8"
server_key_version: "encryptionkey@0"
I20260926 16:32:11.301601 16802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/ts-1/wal/instance:
uuid: "bd748cfeb19d4dd8ac3a8c3407a19b18"
format_stamp: "Formatted at 2026-09-26 16:32:11 on dist-test-slave-khd3"
server_key: "b2bd0f2c15da3c0de82405808ecf2473"
server_key_iv: "5aacc633c793dc2bb7ccba7a874f55f8"
server_key_version: "encryptionkey@0"
I20260926 16:32:11.306756 16802 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260926 16:32:11.310360 16817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:32:11.311900 16802 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260926 16:32:11.312091 16802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/ts-1/wal
uuid: "bd748cfeb19d4dd8ac3a8c3407a19b18"
format_stamp: "Formatted at 2026-09-26 16:32:11 on dist-test-slave-khd3"
server_key: "b2bd0f2c15da3c0de82405808ecf2473"
server_key_iv: "5aacc633c793dc2bb7ccba7a874f55f8"
server_key_version: "encryptionkey@0"
I20260926 16:32:11.312318 16802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:32:11.350342 16802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:32:11.354956 16802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:32:11.355254 16802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:32:11.356289 16802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:32:11.357736 16802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:32:11.357846 16802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:32:11.357995 16802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:32:11.358078 16802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:32:11.390260 16802 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.2:38059
I20260926 16:32:11.390369 16929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.2:38059 every 8 connection(s)
I20260926 16:32:11.391571 16802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
I20260926 16:32:11.395377 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 16802
I20260926 16:32:11.395574 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/ts-1/wal/instance
I20260926 16:32:11.396103 6276 external_mini_cluster.cc:1428] Setting key 989725063ff01627c20e2faaa4e50e59
Sep 26 16:32:11 dist-test-slave-khd3 krb5kdc[16562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440331, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.2@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:32:11.412868 16932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:32:11.394381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:59425 (local address 127.6.33.62:44885)
0926 16:32:11.394799 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0926 16:32:11.394804 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:32:11.396004 (+ 1200us) server_negotiation.cc:407] Connection header received
0926 16:32:11.398154 (+ 2150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:32:11.398159 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:32:11.398254 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:32:11.398381 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:32:11.400065 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:32:11.400861 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:32:11.402428 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:32:11.402856 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:32:11.402994 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:32:11.407068 (+ 4074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:32:11.407073 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:32:11.407076 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:32:11.407111 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:32:11.409795 (+ 2684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:32:11.410517 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:32:11.410525 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:32:11.410529 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:32:11.410660 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:32:11.411089 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:32:11.411096 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:32:11.411099 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:32:11.411380 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:32:11.411541 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 16:32:11.412240 (+ 699us) server_negotiation.cc:299] Negotiation successful
0926 16:32:11.412465 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":209,"threads_started":1}
I20260926 16:32:11.413978 16933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:32:11.395074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:44885 (local address 127.6.33.2:59425)
0926 16:32:11.395843 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0926 16:32:11.395887 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 16:32:11.397812 (+ 1925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:32:11.398679 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:32:11.398689 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:32:11.399292 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:32:11.399884 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:32:11.399902 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:32:11.401018 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:32:11.401022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:32:11.402279 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:32:11.402292 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:32:11.403088 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:32:11.403095 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:32:11.403329 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:32:11.404089 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:32:11.404112 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:32:11.406811 (+ 2699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:32:11.409941 (+ 3130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:32:11.409951 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:32:11.409969 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:32:11.410393 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:32:11.410781 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:32:11.410786 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:32:11.410789 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:32:11.410975 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:32:11.411510 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:32:11.411519 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:32:11.411880 (+ 361us) client_negotiation.cc:769] Sending connection context
0926 16:32:11.412162 (+ 282us) client_negotiation.cc:240] Negotiation successful
0926 16:32:11.412711 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":333,"threads_started":1}
I20260926 16:32:11.415450 16930 heartbeater.cc:347] Connected to a master server at 127.6.33.62:44885
I20260926 16:32:11.415782 16930 heartbeater.cc:464] Registering TS with master...
I20260926 16:32:11.416584 16930 heartbeater.cc:511] Master 127.6.33.62:44885 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.33.3@KRBTEST.COM; defaulting to no policy
I20260926 16:32:11.418171 16607 ts_manager.cc:194] Registered new tserver with Master: bd748cfeb19d4dd8ac3a8c3407a19b18 (127.6.33.2:38059)
I20260926 16:32:11.419019 16607 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.33.2@KRBTEST.COM'} at 127.6.33.2:59425
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab.
Entry for principal kudu/127.6.33.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab.
I20260926 16:32:11.447809 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.3:0
--local_ip_for_outbound_sockets=127.6.33.3
--webserver_interface=127.6.33.3
--webserver_port=0
--tserver_master_addrs=127.6.33.62:44885
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab
--principal=kudu/127.6.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:33815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:32:11.779811 16936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:32:11.780184 16936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:32:11.780366 16936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:32:11.787608 16936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:32:11.787781 16936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:32:11.787978 16936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.3
I20260926 16:32:11.797982 16936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:33815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab
--principal=kudu/127.6.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.33.3
--webserver_port=0
--tserver_master_addrs=127.6.33.62:44885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.3
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:32:11.799402 16936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:32:11.800732 16936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:32:11.814105 16941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:32:11.814281 16942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:32:11.814221 16944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:32:11.815311 16936 server_base.cc:1034] running on GCE node
I20260926 16:32:11.815980 16936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:32:11.817106 16936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:32:11.818302 16936 hybrid_clock.cc:648] HybridClock initialized: now 1790440331818240 us; error 66 us; skew 500 ppm
I20260926 16:32:11.818611 16936 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:32:11 dist-test-slave-khd3 krb5kdc[16562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440331, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:32:11.821781 16936 init.cc:377] Logged in from keytab as kudu/127.6.33.3@KRBTEST.COM (short username kudu)
I20260926 16:32:11.823400 16936 webserver.cc:460] Webserver started at http://127.6.33.3:45891/ using document root <none> and password file <none>
I20260926 16:32:11.824091 16936 fs_manager.cc:359] Metadata directory not provided
I20260926 16:32:11.824249 16936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:32:11.824553 16936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:32:11.827464 16936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/ts-2/data/instance:
uuid: "9562ebf49f8544aba4e5e1de19886a0c"
format_stamp: "Formatted at 2026-09-26 16:32:11 on dist-test-slave-khd3"
server_key: "d537c31c1b37b989d4107ef766caba16"
server_key_iv: "542efdca3b61acedc07ac913e1e2700d"
server_key_version: "encryptionkey@0"
I20260926 16:32:11.828289 16936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/ts-2/wal/instance:
uuid: "9562ebf49f8544aba4e5e1de19886a0c"
format_stamp: "Formatted at 2026-09-26 16:32:11 on dist-test-slave-khd3"
server_key: "d537c31c1b37b989d4107ef766caba16"
server_key_iv: "542efdca3b61acedc07ac913e1e2700d"
server_key_version: "encryptionkey@0"
I20260926 16:32:11.833743 16936 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260926 16:32:11.837692 16951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:32:11.839455 16936 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:32:11.839664 16936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/ts-2/wal
uuid: "9562ebf49f8544aba4e5e1de19886a0c"
format_stamp: "Formatted at 2026-09-26 16:32:11 on dist-test-slave-khd3"
server_key: "d537c31c1b37b989d4107ef766caba16"
server_key_iv: "542efdca3b61acedc07ac913e1e2700d"
server_key_version: "encryptionkey@0"
I20260926 16:32:11.839910 16936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:32:11.857508 16936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:32:11.861811 16936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:32:11.862111 16936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:32:11.863227 16936 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:32:11.864722 16936 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:32:11.864850 16936 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260926 16:32:11.865005 16936 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:32:11.865078 16936 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:32:11.884966 16796 heartbeater.cc:503] Master 127.6.33.62:44885 was elected leader, sending a full tablet report...
I20260926 16:32:11.899020 16936 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.3:36695
I20260926 16:32:11.899039 17063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.3:36695 every 8 connection(s)
I20260926 16:32:11.900332 16936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
I20260926 16:32:11.910583 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 16936
I20260926 16:32:11.910771 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/ts-2/wal/instance
I20260926 16:32:11.911335 6276 external_mini_cluster.cc:1428] Setting key ff1de936311d93a3fe3a54dd4ce0903c
Sep 26 16:32:11 dist-test-slave-khd3 krb5kdc[16562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440331, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.3@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:32:11.921010 16932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:32:11.902924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.3:51575 (local address 127.6.33.62:44885)
0926 16:32:11.903124 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 16:32:11.903130 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:32:11.904964 (+ 1834us) server_negotiation.cc:407] Connection header received
0926 16:32:11.906718 (+ 1754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:32:11.906722 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:32:11.906792 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:32:11.906885 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:32:11.908354 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:32:11.909179 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:32:11.910815 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:32:11.911415 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:32:11.911553 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:32:11.915591 (+ 4038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:32:11.915600 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:32:11.915604 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:32:11.915659 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:32:11.918408 (+ 2749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:32:11.919077 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:32:11.919081 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:32:11.919083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:32:11.919216 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:32:11.919583 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:32:11.919586 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:32:11.919588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:32:11.919835 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:32:11.919936 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 16:32:11.920568 (+ 632us) server_negotiation.cc:299] Negotiation successful
0926 16:32:11.920727 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 16:32:11.922051 17066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:32:11.903476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:44885 (local address 127.6.33.3:51575)
0926 16:32:11.904730 (+ 1254us) negotiation.cc:107] Waiting for socket to connect
0926 16:32:11.904830 (+ 100us) client_negotiation.cc:174] Beginning negotiation
0926 16:32:11.906466 (+ 1636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:32:11.907091 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:32:11.907099 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:32:11.907640 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:32:11.908180 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:32:11.908208 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:32:11.909353 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:32:11.909357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:32:11.910620 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:32:11.910633 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:32:11.911670 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:32:11.911675 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:32:11.911881 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:32:11.912681 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:32:11.912728 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:32:11.915374 (+ 2646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:32:11.918539 (+ 3165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:32:11.918545 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:32:11.918559 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:32:11.918977 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:32:11.919314 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:32:11.919317 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:32:11.919319 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:32:11.919492 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:32:11.919934 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:32:11.919939 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:32:11.920261 (+ 322us) client_negotiation.cc:769] Sending connection context
0926 16:32:11.920488 (+ 227us) client_negotiation.cc:240] Negotiation successful
0926 16:32:11.920963 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":960,"thread_start_us":717,"threads_started":1}
I20260926 16:32:11.923441 17064 heartbeater.cc:347] Connected to a master server at 127.6.33.62:44885
I20260926 16:32:11.923758 17064 heartbeater.cc:464] Registering TS with master...
I20260926 16:32:11.924475 17064 heartbeater.cc:511] Master 127.6.33.62:44885 requested a full tablet report, sending...
I20260926 16:32:11.926020 16607 ts_manager.cc:194] Registered new tserver with Master: 9562ebf49f8544aba4e5e1de19886a0c (127.6.33.3:36695)
I20260926 16:32:11.926862 16607 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.33.3@KRBTEST.COM'} at 127.6.33.3:51575
I20260926 16:32:11.927075 6276 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:32:11.944629 16932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:32:11.932222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48692 (local address 127.6.33.62:44885)
0926 16:32:11.932444 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 16:32:11.932450 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:32:11.932683 (+ 233us) server_negotiation.cc:407] Connection header received
0926 16:32:11.932880 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:32:11.932884 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:32:11.932945 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:32:11.933021 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:32:11.934374 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:32:11.935090 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:32:11.936390 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:32:11.936860 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:32:11.936955 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:32:11.939243 (+ 2288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:32:11.939248 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:32:11.939251 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:32:11.939280 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:32:11.941931 (+ 2651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:32:11.942623 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:32:11.942626 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:32:11.942629 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:32:11.942775 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:32:11.943173 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:32:11.943181 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:32:11.943185 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:32:11.943504 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:32:11.943753 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0926 16:32:11.944066 (+ 313us) server_negotiation.cc:299] Negotiation successful
0926 16:32:11.944341 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 16:32:11.949436 16607 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48692:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:32:11.956271 16607 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:32:11.979902 17077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:32:11.968800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54494 (local address 127.6.33.3:36695)
0926 16:32:11.970491 (+ 1691us) server_negotiation.cc:206] Beginning negotiation
0926 16:32:11.970499 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:32:11.970529 (+ 30us) server_negotiation.cc:407] Connection header received
0926 16:32:11.970640 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:32:11.970649 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:32:11.970923 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:32:11.971828 (+ 905us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:32:11.972153 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:32:11.974429 (+ 2276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:32:11.977167 (+ 2738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:32:11.978521 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:32:11.978764 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:32:11.978967 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0926 16:32:11.979331 (+ 364us) server_negotiation.cc:299] Negotiation successful
0926 16:32:11.979496 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1480,"thread_start_us":1325,"threads_started":1}
I20260926 16:32:11.980105 17076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:32:11.968588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.3:36695 (local address 127.0.0.1:54494)
0926 16:32:11.969602 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0926 16:32:11.969638 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 16:32:11.969792 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:32:11.971170 (+ 1378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:32:11.971179 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:32:11.971221 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:32:11.971631 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:32:11.971645 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:32:11.974613 (+ 2968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:32:11.974627 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:32:11.977000 (+ 2373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:32:11.977015 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:32:11.978944 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:32:11.978951 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:32:11.979137 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:32:11.979162 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 16:32:11.979264 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 16:32:11.979539 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":785,"mutex_wait_us":54,"thread_start_us":632,"threads_started":1}
I20260926 16:32:11.980546 17078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:32:11.970015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.2:38059 (local address 127.0.0.1:46910)
0926 16:32:11.970758 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0926 16:32:11.970781 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 16:32:11.970927 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:32:11.972018 (+ 1091us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:32:11.972023 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:32:11.972055 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:32:11.972529 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:32:11.972540 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:32:11.974254 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:32:11.974261 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:32:11.976875 (+ 2614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:32:11.976886 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:32:11.978769 (+ 1883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:32:11.978775 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:32:11.979027 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:32:11.979122 (+ 95us) client_negotiation.cc:769] Sending connection context
0926 16:32:11.979264 (+ 142us) client_negotiation.cc:240] Negotiation successful
0926 16:32:11.979539 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"mutex_wait_us":98,"thread_start_us":415,"threads_started":1}
I20260926 16:32:11.980829 17079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:32:11.970258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46910 (local address 127.6.33.2:38059)
0926 16:32:11.971402 (+ 1144us) server_negotiation.cc:206] Beginning negotiation
0926 16:32:11.971410 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:32:11.971436 (+ 26us) server_negotiation.cc:407] Connection header received
0926 16:32:11.971567 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:32:11.971575 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:32:11.971812 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:32:11.972793 (+ 981us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:32:11.973027 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:32:11.974098 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:32:11.977024 (+ 2926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:32:11.978521 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:32:11.978665 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:32:11.978888 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0926 16:32:11.979336 (+ 448us) server_negotiation.cc:299] Negotiation successful
0926 16:32:11.979496 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1012,"thread_start_us":921,"threads_started":1}
I20260926 16:32:11.983884 17080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:32:11.970751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:43869 (local address 127.0.0.1:34386)
0926 16:32:11.974805 (+ 4054us) negotiation.cc:107] Waiting for socket to connect
0926 16:32:11.974833 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 16:32:11.974983 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:32:11.975849 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:32:11.975854 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:32:11.975880 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:32:11.976275 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:32:11.976286 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:32:11.977892 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:32:11.977897 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:32:11.981426 (+ 3529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:32:11.981441 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:32:11.983031 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:32:11.983037 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:32:11.983204 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:32:11.983237 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 16:32:11.983352 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 16:32:11.983480 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3887,"mutex_wait_us":33,"thread_start_us":3681,"threads_started":1}
I20260926 16:32:11.983896 17081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:32:11.971299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34386 (local address 127.6.33.1:43869)
0926 16:32:11.973098 (+ 1799us) server_negotiation.cc:206] Beginning negotiation
0926 16:32:11.973109 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:32:11.975010 (+ 1901us) server_negotiation.cc:407] Connection header received
0926 16:32:11.975119 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:32:11.975128 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:32:11.975413 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:32:11.975616 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:32:11.976475 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:32:11.977756 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:32:11.981607 (+ 3851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:32:11.982825 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:32:11.982974 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:32:11.983226 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0926 16:32:11.983402 (+ 176us) server_negotiation.cc:299] Negotiation successful
0926 16:32:11.983560 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":304,"threads_started":1}
I20260926 16:32:11.984408 16999 tablet_service.cc:1467] Processing CreateTablet for tablet e08c3c56806a42a789a39e602632d47a (DEFAULT_TABLE table=test-table [id=44180fa794f94726813c18dbc8757245]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:32:11.984949 16865 tablet_service.cc:1467] Processing CreateTablet for tablet e08c3c56806a42a789a39e602632d47a (DEFAULT_TABLE table=test-table [id=44180fa794f94726813c18dbc8757245]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:32:11.986085 16999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e08c3c56806a42a789a39e602632d47a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:32:11.986263 16865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e08c3c56806a42a789a39e602632d47a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:32:11.986555 16731 tablet_service.cc:1467] Processing CreateTablet for tablet e08c3c56806a42a789a39e602632d47a (DEFAULT_TABLE table=test-table [id=44180fa794f94726813c18dbc8757245]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:32:11.987591 16731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e08c3c56806a42a789a39e602632d47a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:32:11.999037 17082 tablet_bootstrap.cc:492] T e08c3c56806a42a789a39e602632d47a P 944e1cbe75934ca0b6239d3e98ccd5c4: Bootstrap starting.
I20260926 16:32:12.001029 17083 tablet_bootstrap.cc:492] T e08c3c56806a42a789a39e602632d47a P 9562ebf49f8544aba4e5e1de19886a0c: Bootstrap starting.
I20260926 16:32:12.003826 17082 tablet_bootstrap.cc:654] T e08c3c56806a42a789a39e602632d47a P 944e1cbe75934ca0b6239d3e98ccd5c4: Neither blocks nor log segments found. Creating new log.
I20260926 16:32:12.004449 17083 tablet_bootstrap.cc:654] T e08c3c56806a42a789a39e602632d47a P 9562ebf49f8544aba4e5e1de19886a0c: Neither blocks nor log segments found. Creating new log.
I20260926 16:32:12.005285 17082 log.cc:824] T e08c3c56806a42a789a39e602632d47a P 944e1cbe75934ca0b6239d3e98ccd5c4: Log is configured to *not* fsync() on all Append() calls
I20260926 16:32:12.005579 17084 tablet_bootstrap.cc:492] T e08c3c56806a42a789a39e602632d47a P bd748cfeb19d4dd8ac3a8c3407a19b18: Bootstrap starting.
I20260926 16:32:12.005659 17083 log.cc:824] T e08c3c56806a42a789a39e602632d47a P 9562ebf49f8544aba4e5e1de19886a0c: Log is configured to *not* fsync() on all Append() calls
I20260926 16:32:12.009042 17084 tablet_bootstrap.cc:654] T e08c3c56806a42a789a39e602632d47a P bd748cfeb19d4dd8ac3a8c3407a19b18: Neither blocks nor log segments found. Creating new log.
I20260926 16:32:12.009647 17082 tablet_bootstrap.cc:492] T e08c3c56806a42a789a39e602632d47a P 944e1cbe75934ca0b6239d3e98ccd5c4: No bootstrap required, opened a new log
I20260926 16:32:12.009652 17083 tablet_bootstrap.cc:492] T e08c3c56806a42a789a39e602632d47a P 9562ebf49f8544aba4e5e1de19886a0c: No bootstrap required, opened a new log
I20260926 16:32:12.009933 17083 ts_tablet_manager.cc:1397] T e08c3c56806a42a789a39e602632d47a P 9562ebf49f8544aba4e5e1de19886a0c: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260926 16:32:12.009936 17082 ts_tablet_manager.cc:1397] T e08c3c56806a42a789a39e602632d47a P 944e1cbe75934ca0b6239d3e98ccd5c4: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260926 16:32:12.010110 17084 log.cc:824] T e08c3c56806a42a789a39e602632d47a P bd748cfeb19d4dd8ac3a8c3407a19b18: Log is configured to *not* fsync() on all Append() calls
I20260926 16:32:12.013399 17084 tablet_bootstrap.cc:492] T e08c3c56806a42a789a39e602632d47a P bd748cfeb19d4dd8ac3a8c3407a19b18: No bootstrap required, opened a new log
I20260926 16:32:12.013684 17084 ts_tablet_manager.cc:1397] T e08c3c56806a42a789a39e602632d47a P bd748cfeb19d4dd8ac3a8c3407a19b18: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260926 16:32:12.020939 17083 raft_consensus.cc:348] T e08c3c56806a42a789a39e602632d47a P 9562ebf49f8544aba4e5e1de19886a0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9562ebf49f8544aba4e5e1de19886a0c" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 36695 } } peers { permanent_uuid: "bd748cfeb19d4dd8ac3a8c3407a19b18" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 38059 } } peers { permanent_uuid: "944e1cbe75934ca0b6239d3e98ccd5c4" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 43869 } }
I20260926 16:32:12.021168 17082 raft_consensus.cc:348] T e08c3c56806a42a789a39e602632d47a P 944e1cbe75934ca0b6239d3e98ccd5c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9562ebf49f8544aba4e5e1de19886a0c" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 36695 } } peers { permanent_uuid: "bd748cfeb19d4dd8ac3a8c3407a19b18" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 38059 } } peers { permanent_uuid: "944e1cbe75934ca0b6239d3e98ccd5c4" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 43869 } }
I20260926 16:32:12.021198 17084 raft_consensus.cc:348] T e08c3c56806a42a789a39e602632d47a P bd748cfeb19d4dd8ac3a8c3407a19b18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9562ebf49f8544aba4e5e1de19886a0c" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 36695 } } peers { permanent_uuid: "bd748cfeb19d4dd8ac3a8c3407a19b18" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 38059 } } peers { permanent_uuid: "944e1cbe75934ca0b6239d3e98ccd5c4" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 43869 } }
I20260926 16:32:12.021463 17083 raft_consensus.cc:374] T e08c3c56806a42a789a39e602632d47a P 9562ebf49f8544aba4e5e1de19886a0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:32:12.021588 17082 raft_consensus.cc:374] T e08c3c56806a42a789a39e602632d47a P 944e1cbe75934ca0b6239d3e98ccd5c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:32:12.021566 17083 raft_consensus.cc:729] T e08c3c56806a42a789a39e602632d47a P 9562ebf49f8544aba4e5e1de19886a0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9562ebf49f8544aba4e5e1de19886a0c, State: Initialized, Role: FOLLOWER
I20260926 16:32:12.021598 17084 raft_consensus.cc:374] T e08c3c56806a42a789a39e602632d47a P bd748cfeb19d4dd8ac3a8c3407a19b18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:32:12.021684 17082 raft_consensus.cc:729] T e08c3c56806a42a789a39e602632d47a P 944e1cbe75934ca0b6239d3e98ccd5c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 944e1cbe75934ca0b6239d3e98ccd5c4, State: Initialized, Role: FOLLOWER
I20260926 16:32:12.021724 17084 raft_consensus.cc:729] T e08c3c56806a42a789a39e602632d47a P bd748cfeb19d4dd8ac3a8c3407a19b18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd748cfeb19d4dd8ac3a8c3407a19b18, State: Initialized, Role: FOLLOWER
I20260926 16:32:12.022294 17083 consensus_queue.cc:260] T e08c3c56806a42a789a39e602632d47a P 9562ebf49f8544aba4e5e1de19886a0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9562ebf49f8544aba4e5e1de19886a0c" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 36695 } } peers { permanent_uuid: "bd748cfeb19d4dd8ac3a8c3407a19b18" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 38059 } } peers { permanent_uuid: "944e1cbe75934ca0b6239d3e98ccd5c4" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 43869 } }
I20260926 16:32:12.022296 17082 consensus_queue.cc:260] T e08c3c56806a42a789a39e602632d47a P 944e1cbe75934ca0b6239d3e98ccd5c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9562ebf49f8544aba4e5e1de19886a0c" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 36695 } } peers { permanent_uuid: "bd748cfeb19d4dd8ac3a8c3407a19b18" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 38059 } } peers { permanent_uuid: "944e1cbe75934ca0b6239d3e98ccd5c4" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 43869 } }
I20260926 16:32:12.022421 17084 consensus_queue.cc:260] T e08c3c56806a42a789a39e602632d47a P bd748cfeb19d4dd8ac3a8c3407a19b18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9562ebf49f8544aba4e5e1de19886a0c" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 36695 } } peers { permanent_uuid: "bd748cfeb19d4dd8ac3a8c3407a19b18" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 38059 } } peers { permanent_uuid: "944e1cbe75934ca0b6239d3e98ccd5c4" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 43869 } }
I20260926 16:32:12.030143 16930 heartbeater.cc:503] Master 127.6.33.62:44885 was elected leader, sending a full tablet report...
I20260926 16:32:12.030306 17083 ts_tablet_manager.cc:1428] T e08c3c56806a42a789a39e602632d47a P 9562ebf49f8544aba4e5e1de19886a0c: Time spent starting tablet: real 0.020s user 0.010s sys 0.010s
I20260926 16:32:12.030366 17082 ts_tablet_manager.cc:1428] T e08c3c56806a42a789a39e602632d47a P 944e1cbe75934ca0b6239d3e98ccd5c4: Time spent starting tablet: real 0.020s user 0.013s sys 0.007s
I20260926 16:32:12.030762 17084 ts_tablet_manager.cc:1428] T e08c3c56806a42a789a39e602632d47a P bd748cfeb19d4dd8ac3a8c3407a19b18: Time spent starting tablet: real 0.017s user 0.013s sys 0.004s
I20260926 16:32:12.031199 17064 heartbeater.cc:503] Master 127.6.33.62:44885 was elected leader, sending a full tablet report...
I20260926 16:32:12.050704 17090 raft_consensus.cc:482] T e08c3c56806a42a789a39e602632d47a P bd748cfeb19d4dd8ac3a8c3407a19b18 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:32:12.051004 17090 raft_consensus.cc:504] T e08c3c56806a42a789a39e602632d47a P bd748cfeb19d4dd8ac3a8c3407a19b18 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9562ebf49f8544aba4e5e1de19886a0c" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 36695 } } peers { permanent_uuid: "bd748cfeb19d4dd8ac3a8c3407a19b18" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 38059 } } peers { permanent_uuid: "944e1cbe75934ca0b6239d3e98ccd5c4" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 43869 } }
I20260926 16:32:12.052809 17090 leader_election.cc:288] T e08c3c56806a42a789a39e602632d47a P bd748cfeb19d4dd8ac3a8c3407a19b18 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9562ebf49f8544aba4e5e1de19886a0c (127.6.33.3:36695), 944e1cbe75934ca0b6239d3e98ccd5c4 (127.6.33.1:43869)
I20260926 16:32:12.062207 17081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:32:12.054499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:42797 (local address 127.6.33.1:43869)
0926 16:32:12.054704 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 16:32:12.054709 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:32:12.054875 (+ 166us) server_negotiation.cc:407] Connection header received
0926 16:32:12.055022 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:32:12.055026 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:32:12.055083 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:32:12.055173 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:32:12.055863 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:32:12.056928 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:32:12.059697 (+ 2769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:32:12.060973 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:32:12.061114 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:32:12.061222 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 16:32:12.061805 (+ 583us) server_negotiation.cc:299] Negotiation successful
0926 16:32:12.061945 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 16:32:12.062255 17077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:32:12.053164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:58565 (local address 127.6.33.3:36695)
0926 16:32:12.053387 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 16:32:12.053393 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:32:12.054250 (+ 857us) server_negotiation.cc:407] Connection header received
0926 16:32:12.054514 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:32:12.054519 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:32:12.054604 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:32:12.054676 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:32:12.055516 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:32:12.056612 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:32:12.059690 (+ 3078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:32:12.060975 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:32:12.061173 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:32:12.061270 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 16:32:12.061805 (+ 535us) server_negotiation.cc:299] Negotiation successful
0926 16:32:12.061945 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 16:32:12.062714 17092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:32:12.054144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:43869 (local address 127.6.33.2:42797)
0926 16:32:12.054779 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0926 16:32:12.054796 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 16:32:12.054909 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:32:12.055215 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:32:12.055220 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:32:12.055246 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:32:12.055702 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:32:12.055714 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:32:12.057067 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:32:12.057073 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:32:12.059480 (+ 2407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:32:12.059497 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:32:12.061273 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:32:12.061280 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:32:12.061602 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:32:12.061649 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 16:32:12.061751 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 16:32:12.061984 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"mutex_wait_us":254,"thread_start_us":318,"threads_started":1}
I20260926 16:32:12.062742 17091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:32:12.053247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.3:36695 (local address 127.6.33.2:58565)
0926 16:32:12.054136 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0926 16:32:12.054158 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 16:32:12.054310 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:32:12.054883 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:32:12.054887 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:32:12.054926 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:32:12.055349 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:32:12.055366 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:32:12.056822 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:32:12.056826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:32:12.059480 (+ 2654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:32:12.059497 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:32:12.061275 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:32:12.061280 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:32:12.061583 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:32:12.061646 (+ 63us) client_negotiation.cc:769] Sending connection context
0926 16:32:12.061751 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 16:32:12.061984 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":694,"mutex_wait_us":213,"spinlock_wait_cycles":1408,"thread_start_us":472,"threads_started":1}
I20260926 16:32:12.063833 16751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e08c3c56806a42a789a39e602632d47a" candidate_uuid: "bd748cfeb19d4dd8ac3a8c3407a19b18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "944e1cbe75934ca0b6239d3e98ccd5c4" is_pre_election: true
I20260926 16:32:12.063838 17019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e08c3c56806a42a789a39e602632d47a" candidate_uuid: "bd748cfeb19d4dd8ac3a8c3407a19b18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9562ebf49f8544aba4e5e1de19886a0c" is_pre_election: true
I20260926 16:32:12.064397 16751 raft_consensus.cc:2454] T e08c3c56806a42a789a39e602632d47a P 944e1cbe75934ca0b6239d3e98ccd5c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd748cfeb19d4dd8ac3a8c3407a19b18 in term 0.
I20260926 16:32:12.064397 17019 raft_consensus.cc:2454] T e08c3c56806a42a789a39e602632d47a P 9562ebf49f8544aba4e5e1de19886a0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd748cfeb19d4dd8ac3a8c3407a19b18 in term 0.
I20260926 16:32:12.065062 16819 leader_election.cc:302] T e08c3c56806a42a789a39e602632d47a P bd748cfeb19d4dd8ac3a8c3407a19b18 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 944e1cbe75934ca0b6239d3e98ccd5c4, bd748cfeb19d4dd8ac3a8c3407a19b18; no voters:
I20260926 16:32:12.065475 17090 raft_consensus.cc:2781] T e08c3c56806a42a789a39e602632d47a P bd748cfeb19d4dd8ac3a8c3407a19b18 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:32:12.065639 17090 raft_consensus.cc:482] T e08c3c56806a42a789a39e602632d47a P bd748cfeb19d4dd8ac3a8c3407a19b18 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:32:12.065749 17090 raft_consensus.cc:3037] T e08c3c56806a42a789a39e602632d47a P bd748cfeb19d4dd8ac3a8c3407a19b18 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:32:12.067922 17090 raft_consensus.cc:504] T e08c3c56806a42a789a39e602632d47a P bd748cfeb19d4dd8ac3a8c3407a19b18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9562ebf49f8544aba4e5e1de19886a0c" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 36695 } } peers { permanent_uuid: "bd748cfeb19d4dd8ac3a8c3407a19b18" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 38059 } } peers { permanent_uuid: "944e1cbe75934ca0b6239d3e98ccd5c4" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 43869 } }
I20260926 16:32:12.068534 17090 leader_election.cc:288] T e08c3c56806a42a789a39e602632d47a P bd748cfeb19d4dd8ac3a8c3407a19b18 [CANDIDATE]: Term 1 election: Requested vote from peers 9562ebf49f8544aba4e5e1de19886a0c (127.6.33.3:36695), 944e1cbe75934ca0b6239d3e98ccd5c4 (127.6.33.1:43869)
I20260926 16:32:12.068997 17019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e08c3c56806a42a789a39e602632d47a" candidate_uuid: "bd748cfeb19d4dd8ac3a8c3407a19b18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9562ebf49f8544aba4e5e1de19886a0c"
I20260926 16:32:12.069180 17019 raft_consensus.cc:3037] T e08c3c56806a42a789a39e602632d47a P 9562ebf49f8544aba4e5e1de19886a0c [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:32:12.069159 16751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e08c3c56806a42a789a39e602632d47a" candidate_uuid: "bd748cfeb19d4dd8ac3a8c3407a19b18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "944e1cbe75934ca0b6239d3e98ccd5c4"
I20260926 16:32:12.069381 16751 raft_consensus.cc:3037] T e08c3c56806a42a789a39e602632d47a P 944e1cbe75934ca0b6239d3e98ccd5c4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:32:12.071416 17019 raft_consensus.cc:2454] T e08c3c56806a42a789a39e602632d47a P 9562ebf49f8544aba4e5e1de19886a0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd748cfeb19d4dd8ac3a8c3407a19b18 in term 1.
I20260926 16:32:12.071481 16751 raft_consensus.cc:2454] T e08c3c56806a42a789a39e602632d47a P 944e1cbe75934ca0b6239d3e98ccd5c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd748cfeb19d4dd8ac3a8c3407a19b18 in term 1.
I20260926 16:32:12.071851 16819 leader_election.cc:302] T e08c3c56806a42a789a39e602632d47a P bd748cfeb19d4dd8ac3a8c3407a19b18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9562ebf49f8544aba4e5e1de19886a0c, bd748cfeb19d4dd8ac3a8c3407a19b18; no voters:
I20260926 16:32:12.072144 17090 raft_consensus.cc:2781] T e08c3c56806a42a789a39e602632d47a P bd748cfeb19d4dd8ac3a8c3407a19b18 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:32:12.072679 17090 raft_consensus.cc:686] T e08c3c56806a42a789a39e602632d47a P bd748cfeb19d4dd8ac3a8c3407a19b18 [term 1 LEADER]: Becoming Leader. State: Replica: bd748cfeb19d4dd8ac3a8c3407a19b18, State: Running, Role: LEADER
I20260926 16:32:12.073275 17090 consensus_queue.cc:237] T e08c3c56806a42a789a39e602632d47a P bd748cfeb19d4dd8ac3a8c3407a19b18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9562ebf49f8544aba4e5e1de19886a0c" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 36695 } } peers { permanent_uuid: "bd748cfeb19d4dd8ac3a8c3407a19b18" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 38059 } } peers { permanent_uuid: "944e1cbe75934ca0b6239d3e98ccd5c4" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 43869 } }
I20260926 16:32:12.077975 16607 catalog_manager.cc:5495] T e08c3c56806a42a789a39e602632d47a P bd748cfeb19d4dd8ac3a8c3407a19b18 reported cstate change: term changed from 0 to 1, leader changed from <none> to bd748cfeb19d4dd8ac3a8c3407a19b18 (127.6.33.2). New cstate: current_term: 1 leader_uuid: "bd748cfeb19d4dd8ac3a8c3407a19b18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9562ebf49f8544aba4e5e1de19886a0c" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 36695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd748cfeb19d4dd8ac3a8c3407a19b18" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 38059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "944e1cbe75934ca0b6239d3e98ccd5c4" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 43869 } health_report { overall_health: UNKNOWN } } }
W20260926 16:32:12.101713 16797 tablet.cc:2369] T e08c3c56806a42a789a39e602632d47a P 944e1cbe75934ca0b6239d3e98ccd5c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:32:12.110496 17079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:32:12.105073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46922 (local address 127.6.33.2:38059)
0926 16:32:12.105279 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 16:32:12.105283 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:32:12.105300 (+ 17us) server_negotiation.cc:407] Connection header received
0926 16:32:12.105453 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:32:12.105456 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:32:12.105537 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:32:12.105668 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:32:12.106140 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:32:12.106846 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:32:12.108170 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:32:12.108582 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:32:12.108677 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:32:12.109245 (+ 568us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:32:12.109801 (+ 556us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:32:12.109922 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 16:32:12.110072 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 16:32:12.110188 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260926 16:32:12.118971 16751 raft_consensus.cc:1261] T e08c3c56806a42a789a39e602632d47a P 944e1cbe75934ca0b6239d3e98ccd5c4 [term 1 FOLLOWER]: Refusing update from remote peer bd748cfeb19d4dd8ac3a8c3407a19b18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:32:12.119030 17019 raft_consensus.cc:1261] T e08c3c56806a42a789a39e602632d47a P 9562ebf49f8544aba4e5e1de19886a0c [term 1 FOLLOWER]: Refusing update from remote peer bd748cfeb19d4dd8ac3a8c3407a19b18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:32:12.120033 17093 consensus_queue.cc:1035] T e08c3c56806a42a789a39e602632d47a P bd748cfeb19d4dd8ac3a8c3407a19b18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9562ebf49f8544aba4e5e1de19886a0c" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 36695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:32:12.120525 17090 consensus_queue.cc:1035] T e08c3c56806a42a789a39e602632d47a P bd748cfeb19d4dd8ac3a8c3407a19b18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "944e1cbe75934ca0b6239d3e98ccd5c4" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 43869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:32:12.136502 17098 mvcc.cc:205] Tried to move back new op lower bound from 7333643600348700672 to 7333643600175833088. Current Snapshot: MvccSnapshot[applied={T|T < 7333643600348700672}]
I20260926 16:32:12.147853 16607 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:32:12.148121 16607 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:32:12.151916 16607 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c64454921d8049efa0c88749678424be: Sending DeleteTablet for 3 replicas of tablet e08c3c56806a42a789a39e602632d47a
I20260926 16:32:12.153337 16731 tablet_service.cc:1514] Processing DeleteTablet for tablet e08c3c56806a42a789a39e602632d47a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:32:12 UTC) from {username='kudu', principal='kudu/127.6.33.62@KRBTEST.COM'} at 127.0.0.1:34386
I20260926 16:32:12.153596 16999 tablet_service.cc:1514] Processing DeleteTablet for tablet e08c3c56806a42a789a39e602632d47a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:32:12 UTC) from {username='kudu', principal='kudu/127.6.33.62@KRBTEST.COM'} at 127.0.0.1:54494
I20260926 16:32:12.154583 16865 tablet_service.cc:1514] Processing DeleteTablet for tablet e08c3c56806a42a789a39e602632d47a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:32:12 UTC) from {username='kudu', principal='kudu/127.6.33.62@KRBTEST.COM'} at 127.0.0.1:46910
I20260926 16:32:12.155751 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 16668
I20260926 16:32:12.170987 17108 tablet_replica.cc:330] T e08c3c56806a42a789a39e602632d47a P bd748cfeb19d4dd8ac3a8c3407a19b18: stopping tablet replica
I20260926 16:32:12.171819 17108 raft_consensus.cc:2229] T e08c3c56806a42a789a39e602632d47a P bd748cfeb19d4dd8ac3a8c3407a19b18 [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:32:12.172887 17108 raft_consensus.cc:2258] T e08c3c56806a42a789a39e602632d47a P bd748cfeb19d4dd8ac3a8c3407a19b18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:32:12.175812 17108 ts_tablet_manager.cc:1905] T e08c3c56806a42a789a39e602632d47a P bd748cfeb19d4dd8ac3a8c3407a19b18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:32:12.176549 17107 tablet_replica.cc:330] T e08c3c56806a42a789a39e602632d47a P 9562ebf49f8544aba4e5e1de19886a0c: stopping tablet replica
Sep 26 16:32:12 dist-test-slave-khd3 krb5kdc[16562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.1@KRBTEST.COM
I20260926 16:32:12.176774 17106 tablet_replica.cc:330] T e08c3c56806a42a789a39e602632d47a P 944e1cbe75934ca0b6239d3e98ccd5c4: stopping tablet replica
I20260926 16:32:12.177176 17106 raft_consensus.cc:2229] T e08c3c56806a42a789a39e602632d47a P 944e1cbe75934ca0b6239d3e98ccd5c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:32:12.177186 17107 raft_consensus.cc:2229] T e08c3c56806a42a789a39e602632d47a P 9562ebf49f8544aba4e5e1de19886a0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:32:12.177698 17107 raft_consensus.cc:2258] T e08c3c56806a42a789a39e602632d47a P 9562ebf49f8544aba4e5e1de19886a0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:32:12.177712 17106 raft_consensus.cc:2258] T e08c3c56806a42a789a39e602632d47a P 944e1cbe75934ca0b6239d3e98ccd5c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:32:12.181115 17107 ts_tablet_manager.cc:1905] T e08c3c56806a42a789a39e602632d47a P 9562ebf49f8544aba4e5e1de19886a0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:32:12.184060 17106 ts_tablet_manager.cc:1905] T e08c3c56806a42a789a39e602632d47a P 944e1cbe75934ca0b6239d3e98ccd5c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:32:12.186000 17108 ts_tablet_manager.cc:1918] T e08c3c56806a42a789a39e602632d47a P bd748cfeb19d4dd8ac3a8c3407a19b18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:32:12.186439 17108 log.cc:1196] T e08c3c56806a42a789a39e602632d47a P bd748cfeb19d4dd8ac3a8c3407a19b18: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/ts-1/wal/wals/e08c3c56806a42a789a39e602632d47a
I20260926 16:32:12.187350 17108 ts_tablet_manager.cc:1939] T e08c3c56806a42a789a39e602632d47a P bd748cfeb19d4dd8ac3a8c3407a19b18: Deleting consensus metadata
I20260926 16:32:12.187899 17107 ts_tablet_manager.cc:1918] T e08c3c56806a42a789a39e602632d47a P 9562ebf49f8544aba4e5e1de19886a0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:32:12.188053 17107 log.cc:1196] T e08c3c56806a42a789a39e602632d47a P 9562ebf49f8544aba4e5e1de19886a0c: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440233850748-6276-0/minicluster-data/ts-2/wal/wals/e08c3c56806a42a789a39e602632d47a
I20260926 16:32:12.188553 17107 ts_tablet_manager.cc:1939] T e08c3c56806a42a789a39e602632d47a P 9562ebf49f8544aba4e5e1de19886a0c: Deleting consensus metadata
I20260926 16:32:12.189929 16592 catalog_manager.cc:4841] TS bd748cfeb19d4dd8ac3a8c3407a19b18 (127.6.33.2:38059): tablet e08c3c56806a42a789a39e602632d47a (table test-table [id=44180fa794f94726813c18dbc8757245]) successfully deleted
I20260926 16:32:12.190294 16592 catalog_manager.cc:4841] TS 9562ebf49f8544aba4e5e1de19886a0c (127.6.33.3:36695): tablet e08c3c56806a42a789a39e602632d47a (table test-table [id=44180fa794f94726813c18dbc8757245]) successfully deleted
I20260926 16:32:12.192013 17081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:32:12.157197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34398 (local address 127.6.33.1:43869)
0926 16:32:12.157424 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 16:32:12.157433 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:32:12.161792 (+ 4359us) server_negotiation.cc:407] Connection header received
0926 16:32:12.162644 (+ 852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:32:12.162650 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:32:12.162769 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:32:12.162845 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:32:12.166357 (+ 3512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:32:12.167532 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:32:12.171347 (+ 3815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:32:12.172039 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:32:12.172141 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:32:12.177976 (+ 5835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:32:12.177994 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:32:12.178012 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:32:12.178085 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:32:12.182455 (+ 4370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:32:12.188784 (+ 6329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:32:12.188795 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:32:12.188799 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:32:12.188990 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:32:12.189524 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:32:12.189530 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:32:12.189533 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:32:12.190003 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:32:12.190292 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0926 16:32:12.190898 (+ 606us) server_negotiation.cc:299] Negotiation successful
0926 16:32:12.191622 (+ 724us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 16:32:12.192796 16791 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:32:12.283443 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 16668
W20260926 16:32:12.300369 16592 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.33.1:43869 (error 108)
I20260926 16:32:12.301998 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 16802
I20260926 16:32:12.302338 17080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:32:12.301877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:43869 (local address 127.0.0.1:34402)
0926 16:32:12.302049 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 16:32:12.302195 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.1:43869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 16:32:12.302667 16592 catalog_manager.cc:4584] TS 944e1cbe75934ca0b6239d3e98ccd5c4 (127.6.33.1:43869): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e08c3c56806a42a789a39e602632d47a: Network error: Client connection negotiation failed: client connection to 127.6.33.1:43869: connect: Connection refused (error 111)
Sep 26 16:32:12 dist-test-slave-khd3 krb5kdc[16562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.2@KRBTEST.COM
I20260926 16:32:12.324091 17079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:32:12.302933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46930 (local address 127.6.33.2:38059)
0926 16:32:12.303106 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 16:32:12.303110 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:32:12.303127 (+ 17us) server_negotiation.cc:407] Connection header received
0926 16:32:12.303180 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:32:12.303183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:32:12.303241 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:32:12.303330 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:32:12.304964 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:32:12.305888 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:32:12.307912 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:32:12.308827 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:32:12.308991 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:32:12.314452 (+ 5461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:32:12.314470 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:32:12.314490 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:32:12.314536 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:32:12.320670 (+ 6134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:32:12.321614 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:32:12.321623 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:32:12.321627 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:32:12.321830 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:32:12.322387 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:32:12.322392 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:32:12.322396 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:32:12.322811 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:32:12.323148 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0926 16:32:12.323445 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 16:32:12.323680 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 16:32:12.324889 16925 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:32:12.356356 17080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:32:12.355753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:43869 (local address 127.0.0.1:34416)
0926 16:32:12.355986 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0926 16:32:12.356119 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.1:43869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 16:32:12.357618 17080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:32:12.357189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:43869 (local address 127.0.0.1:34428)
0926 16:32:12.357347 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 16:32:12.357462 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.1:43869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 16:32:12.418476 17080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:32:12.417899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:43869 (local address 127.0.0.1:34442)
0926 16:32:12.418204 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0926 16:32:12.418301 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.1:43869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260926 16:32:12.419569 17080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:32:12.419220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:43869 (local address 127.0.0.1:34456)
0926 16:32:12.419362 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 16:32:12.419456 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.1:43869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 16:32:12.450733 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 16802
I20260926 16:32:12.469543 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 16936
Sep 26 16:32:12 dist-test-slave-khd3 krb5kdc[16562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.3@KRBTEST.COM
I20260926 16:32:12.484026 17077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:32:12.470250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54500 (local address 127.6.33.3:36695)
0926 16:32:12.470443 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 16:32:12.470448 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:32:12.470465 (+ 17us) server_negotiation.cc:407] Connection header received
0926 16:32:12.470613 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:32:12.470617 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:32:12.470678 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:32:12.470769 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:32:12.471953 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:32:12.472651 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:32:12.474012 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:32:12.474388 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:32:12.474475 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:32:12.478842 (+ 4367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:32:12.478855 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:32:12.478870 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:32:12.478899 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:32:12.481772 (+ 2873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:32:12.482375 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:32:12.482382 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:32:12.482384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:32:12.482517 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:32:12.482805 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:32:12.482808 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:32:12.482810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:32:12.483119 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:32:12.483331 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0926 16:32:12.483526 (+ 195us) server_negotiation.cc:299] Negotiation successful
0926 16:32:12.483674 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 16:32:12.484647 17059 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:32:12.500551 17080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:32:12.500029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:43869 (local address 127.0.0.1:34458)
0926 16:32:12.500259 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0926 16:32:12.500366 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.1:43869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260926 16:32:12.501734 17080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:32:12.501314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:43869 (local address 127.0.0.1:34472)
0926 16:32:12.501516 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 16:32:12.501597 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.1:43869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260926 16:32:12.574713 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 16936
I20260926 16:32:12.591475 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 16575
I20260926 16:32:12.592622 16637 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:32:12.701017 17080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:32:12.700437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:43869 (local address 127.0.0.1:34474)
0926 16:32:12.700661 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0926 16:32:12.700807 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.1:43869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260926 16:32:12.701046 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 16575
2026-09-26T16:32:12Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4297 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:32:12 dist-test-slave-khd3 krb5kdc[17117](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:32:12 dist-test-slave-khd3 krb5kdc[17117](info): set up 2 sockets
Sep 26 16:32:12 dist-test-slave-khd3 krb5kdc[17117](info): commencing operation
krb5kdc: starting...
W20260926 16:32:13.806631 6276 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Sep 26 16:32:13 dist-test-slave-khd3 krb5kdc[17117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:32:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:32:13Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.6.33.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.6.33.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.6.33.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/krb5kdc/rangeradmin_127.6.33.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.6.33.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/krb5kdc/rangeradmin_127.6.33.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.6.33.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.6.33.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.6.33.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/krb5kdc/rangerlookup_127.6.33.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.6.33.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/krb5kdc/rangerlookup_127.6.33.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.33.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.33.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.33.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/krb5kdc/HTTP_127.6.33.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.33.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/krb5kdc/HTTP_127.6.33.20@KRBTEST.COM.keytab.
I20260926 16:32:14.135936 6276 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskVxavWi/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/postgres -l logfile start
2026-09-26 16:32:16.779 UTC [17147] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 16:32:16.779 UTC [17147] LOG: listening on IPv4 address "127.6.33.20", port 36891
2026-09-26 16:32:16.783 UTC [17147] LOG: listening on Unix socket "/tmp/.s.PGSQL.36891"
2026-09-26 16:32:16.806 UTC [17150] LOG: database system was shut down at 2026-09-26 16:32:15 UTC
2026-09-26 16:32:16.810 UTC [17147] LOG: database system is ready to accept connections
I20260926 16:32:17.794386 6276 mini_postgres.cc:96] Postgres bound to 36891
2026-09-26 16:32:17.812 UTC [17158] FATAL: database "slave" does not exist
127.6.33.20:36891 - accepting connections
I20260926 16:32:17.812959 6276 mini_ranger.cc:161] Starting Ranger...
I20260926 16:32:17.840039 6276 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 16:32:18.361445 6276 mini_ranger.cc:87] Created ranger Postgres database
I20260926 16:32:18.361613 6276 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ranger-admin
2026-09-26 16:32:18,668 [I] DB FLAVOR :POSTGRES
2026-09-26 16:32:18,668 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 16:32:18,669 [I] Checking connection..
2026-09-26 16:32:18,669 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxavWi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVxavWi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.33.20:36891/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 16:32:18,944 [I] Checking connection passed.
2026-09-26 16:32:18,945 [I] --------- Verifying version history table ---------
2026-09-26 16:32:18,945 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxavWi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVxavWi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.33.20:36891/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 16:32:19,202 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 16:32:19,202 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxavWi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVxavWi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.33.20:36891/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 16:32:19,470 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 16:32:19,470 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 16:32:19,470 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxavWi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVxavWi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.33.20:36891/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskVxavWi/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 16:32:19.733 UTC [17222] WARNING: there is no transaction in progress
2026-09-26 16:32:19,740 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 16:32:19,740 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxavWi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVxavWi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.33.20:36891/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 16:32:19,989 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 16:32:19,989 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 16:32:19,989 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxavWi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVxavWi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.33.20:36891/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 16:32:20,243 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxavWi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVxavWi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.33.20:36891/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 16:32:20,512 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxavWi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVxavWi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.33.20:36891/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-khd3.c.gcp-upstream.internal','N') ;"
2026-09-26 16:32:20,781 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxavWi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVxavWi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.33.20:36891/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 16:32:21,035 [I] Table x_portal_user does not exist in database ranger
2026-09-26 16:32:21,036 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxavWi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVxavWi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.33.20:36891/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 16:32:21,295 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 16:32:21,296 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxavWi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVxavWi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.33.20:36891/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 16:32:21,583 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxavWi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVxavWi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.33.20:36891/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 16:32:21,829 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 16:32:21,830 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxavWi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVxavWi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.33.20:36891/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskVxavWi/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 16:32:22.855 UTC [17330] WARNING: there is no transaction in progress
2026-09-26 16:32:22.875 UTC [17330] WARNING: there is no transaction in progress
2026-09-26 16:32:22.893 UTC [17330] WARNING: there is no transaction in progress
2026-09-26 16:32:22.909 UTC [17330] WARNING: there is no transaction in progress
2026-09-26 16:32:22.928 UTC [17330] WARNING: there is no transaction in progress
2026-09-26 16:32:22.947 UTC [17330] WARNING: there is no transaction in progress
2026-09-26 16:32:23.004 UTC [17330] WARNING: there is no transaction in progress
2026-09-26 16:32:23.012 UTC [17330] WARNING: there is no transaction in progress
2026-09-26 16:32:23.028 UTC [17330] WARNING: there is no transaction in progress
2026-09-26 16:32:23.040 UTC [17330] WARNING: there is no transaction in progress
2026-09-26 16:32:23.051 UTC [17330] WARNING: there is no transaction in progress
2026-09-26 16:32:23.067 UTC [17330] WARNING: there is no transaction in progress
2026-09-26 16:32:23.078 UTC [17330] WARNING: there is no transaction in progress
2026-09-26 16:32:23.085 UTC [17330] WARNING: there is no transaction in progress
2026-09-26 16:32:23.093 UTC [17330] WARNING: there is no transaction in progress
2026-09-26 16:32:23.100 UTC [17330] WARNING: there is no transaction in progress
2026-09-26 16:32:23,802 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 16:32:23,802 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxavWi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVxavWi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.33.20:36891/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 16:32:24,058 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 16:32:24,058 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxavWi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVxavWi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.33.20:36891/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 16:32:24,303 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 16:32:24,303 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxavWi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVxavWi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.33.20:36891/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 16:32:24,538 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 16:32:24,538 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxavWi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVxavWi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.33.20:36891/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 16:32:24,811 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxavWi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVxavWi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.33.20:36891/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 16:32:25,073 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxavWi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVxavWi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.33.20:36891/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-khd3.c.gcp-upstream.internal';"
2026-09-26 16:32:25,309 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 16:32:25,310 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxavWi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVxavWi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.33.20:36891/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 16:32:25,557 [I] DB_PATCHES have already been applied
I20260926 16:32:25.563499 6276 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ranger-admin:/tmp/dist-test-taskVxavWi/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskVxavWi/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskVxavWi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVxavWi/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 16:32:25.563619 6276 mini_ranger.cc:193] Using host: 127.6.33.20
I20260926 16:32:25.578997 6276 mini_ranger.cc:227] Ranger admin URL: http://127.6.33.20:47001
Sep 26, 2026 4:32:25 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 4:32:25 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 4:32:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 4:32:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 4:32:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 4:32:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 4:32:26 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 16:32:26.715308 6276 mini_ranger.cc:160] Time spent starting Ranger: real 8.902s user 0.000s sys 0.040s
Sep 26, 2026 4:32:48 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 4:32:48 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 4:32:48 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 4:32:48 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 4:32:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 4:32:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 4:32:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 4:32:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 4:32:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 4:32:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 4:32:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 4:32:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 4:32:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 4:32:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 4:32:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 4:32:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 4:32:48 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 4:32:49 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 16:32:49.902956 6276 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.6.33.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.6.33.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.6.33.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/krb5kdc/rangerkms_127.6.33.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.6.33.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/krb5kdc/rangerkms_127.6.33.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.33.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.6.33.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.33.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/krb5kdc/HTTP_127.6.33.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.33.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/krb5kdc/HTTP_127.6.33.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 16:32:50 dist-test-slave-khd3 krb5kdc[17117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440370, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 16:32:50.078842 6276 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 16:32:50.100343 6276 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 16:32:51.102469 6276 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 16:32:51.102593 6276 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ranger-kms
I20260926 16:32:51.102770 6276 mini_ranger_kms.cc:227] Using postgres at 127.6.33.20:36891
2026-09-26 16:32:51,521 [I] DB FLAVOR :POSTGRES
2026-09-26 16:32:51,521 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 16:32:51,521 [I] Checking connection
2026-09-26 16:32:51,521 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxavWi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.33.20:36891/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 16:32:51,765 [I] connection success
2026-09-26 16:32:51,765 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 16:32:51,765 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 16:32:51,765 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxavWi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.33.20:36891/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 16:32:52,002 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 16:32:52,002 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 16:32:52,002 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 16:32:52,003 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxavWi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.33.20:36891/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 16:32:52,261 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 16:32:52.809324 6276 mini_ranger_kms.cc:310] Created kms service
I20260926 16:32:53.018151 6276 mini_ranger_kms.cc:326] Created kudu user
I20260926 16:32:53.064442 6276 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 16:32:53 dist-test-slave-khd3 krb5kdc[17117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440370, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.6.33.20@KRBTEST.COM
I20260926 16:32:53.463820 6276 mini_ranger_kms.cc:384] Added ranger policy
I20260926 16:32:53.464130 6276 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ranger-kms:/tmp/dist-test-taskVxavWi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVxavWi/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskVxavWi/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 16:32:53.464197 6276 mini_ranger_kms.cc:240] Using host: 127.6.33.20
I20260926 16:32:53.489733 6276 mini_ranger_kms.cc:276] Ranger KMS PID: 17511
I20260926 16:32:53.489935 6276 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.6.33.20:40679
Sep 26, 2026 4:32:54 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 4:32:54 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 4:32:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 4:32:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 4:32:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 4:32:55 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 26 16:32:55 dist-test-slave-khd3 krb5kdc[17117](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790440375, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.33.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:32:55.796314 6276 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.717s user 0.113s sys 0.325s
I20260926 16:32:55.796591 6276 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 16:32:55.796836 6276 mini_ranger_kms.cc:401] 127.6.33.20:40679/kms/v1/keys
Sep 26 16:32:56 dist-test-slave-khd3 krb5kdc[17117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440375, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.33.20@KRBTEST.COM for HTTP/127.6.33.20@KRBTEST.COM
Sep 26 16:32:56 dist-test-slave-khd3 krb5kdc[17117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440375, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.33.20@KRBTEST.COM for HTTP/127.6.33.20@KRBTEST.COM
Sep 26 16:32:58 dist-test-slave-khd3 krb5kdc[17117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.6.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab.
Entry for principal kudu/127.6.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab.
I20260926 16:32:58.785405 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:39201
--webserver_interface=127.6.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:43483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.33.62:39201
--ranger_config_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.33.20:40679/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:32:59.121629 17561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:32:59.121991 17561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:32:59.122143 17561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:32:59.129457 17561 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:32:59.129606 17561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:32:59.129701 17561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:32:59.129807 17561 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:32:59.129887 17561 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:32:59.140410 17561 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:43483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/master-0/wal
--ranger_kms_url=127.6.33.20:40679/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.6.33.62:39201
--ranger_config_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:39201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:32:59.141829 17561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:32:59.143143 17561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:32:59.155699 17569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:32:59.155637 17567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:32:59.155637 17566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:32:59.156834 17561 server_base.cc:1034] running on GCE node
I20260926 16:32:59.157655 17561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:32:59.159240 17561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:32:59.160468 17561 hybrid_clock.cc:648] HybridClock initialized: now 1790440379160430 us; error 36 us; skew 500 ppm
I20260926 16:32:59.160900 17561 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:32:59 dist-test-slave-khd3 krb5kdc[17117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440379, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:32:59.173581 17561 init.cc:377] Logged in from keytab as kudu/127.6.33.62@KRBTEST.COM (short username kudu)
I20260926 16:32:59.175334 17561 webserver.cc:460] Webserver started at http://127.6.33.62:40951/ using document root <none> and password file <none>
I20260926 16:32:59.176069 17561 fs_manager.cc:359] Metadata directory not provided
I20260926 16:32:59.176254 17561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:32:59.176581 17561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 16:32:59 dist-test-slave-khd3 krb5kdc[17117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440379, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.62@KRBTEST.COM for HTTP/127.6.33.20@KRBTEST.COM
I20260926 16:32:59.237705 17561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/master-0/data/instance:
uuid: "f34b2bc957b8474fa4bcb5ff02ffd06b"
format_stamp: "Formatted at 2026-09-26 16:32:59 on dist-test-slave-khd3"
server_key: "0250a7eae6036a66a7e4d661a00c6ca9"
server_key_iv: "a4ef44ce6e21e591357a5bf70e965aa5"
server_key_version: "kuduclusterkey@0"
I20260926 16:32:59.238695 17561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/master-0/wal/instance:
uuid: "f34b2bc957b8474fa4bcb5ff02ffd06b"
format_stamp: "Formatted at 2026-09-26 16:32:59 on dist-test-slave-khd3"
server_key: "0250a7eae6036a66a7e4d661a00c6ca9"
server_key_iv: "a4ef44ce6e21e591357a5bf70e965aa5"
server_key_version: "kuduclusterkey@0"
I20260926 16:32:59.244535 17561 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 16:32:59.261897 17576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:32:59.263605 17561 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260926 16:32:59.263803 17561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/master-0/wal
uuid: "f34b2bc957b8474fa4bcb5ff02ffd06b"
format_stamp: "Formatted at 2026-09-26 16:32:59 on dist-test-slave-khd3"
server_key: "0250a7eae6036a66a7e4d661a00c6ca9"
server_key_iv: "a4ef44ce6e21e591357a5bf70e965aa5"
server_key_version: "kuduclusterkey@0"
I20260926 16:32:59.264078 17561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:32:59.294416 17561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:32:59.299203 17561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:32:59.299530 17561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:32:59.319249 17561 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.62:39201
I20260926 16:32:59.319285 17627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.62:39201 every 8 connection(s)
I20260926 16:32:59.320647 17561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
I20260926 16:32:59.324416 17628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:32:59.325627 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 17561
I20260926 16:32:59.325835 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/master-0/wal/instance
I20260926 16:32:59.326457 6276 external_mini_cluster.cc:1428] Setting key 287a8dc0cc29404c8dcefc4b8a264683
I20260926 16:32:59.338492 17628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f34b2bc957b8474fa4bcb5ff02ffd06b: Bootstrap starting.
Sep 26 16:32:59 dist-test-slave-khd3 krb5kdc[17117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:32:59.342522 17628 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f34b2bc957b8474fa4bcb5ff02ffd06b: Neither blocks nor log segments found. Creating new log.
I20260926 16:32:59.343644 17628 log.cc:824] T 00000000000000000000000000000000 P f34b2bc957b8474fa4bcb5ff02ffd06b: Log is configured to *not* fsync() on all Append() calls
I20260926 16:32:59.347059 17628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f34b2bc957b8474fa4bcb5ff02ffd06b: No bootstrap required, opened a new log
I20260926 16:32:59.349570 17631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:32:59.328436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43882 (local address 127.6.33.62:39201)
0926 16:32:59.329266 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0926 16:32:59.329281 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 16:32:59.329327 (+ 46us) server_negotiation.cc:407] Connection header received
0926 16:32:59.330895 (+ 1568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:32:59.330916 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:32:59.331320 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:32:59.331657 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:32:59.333252 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:32:59.334533 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:32:59.336337 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:32:59.337132 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:32:59.337264 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:32:59.341897 (+ 4633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:32:59.341911 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:32:59.341939 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:32:59.341983 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:32:59.345104 (+ 3121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:32:59.345978 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:32:59.345985 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:32:59.345991 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:32:59.346171 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:32:59.346627 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:32:59.346630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:32:59.346633 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:32:59.347238 (+ 605us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:32:59.347465 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0926 16:32:59.347972 (+ 507us) server_negotiation.cc:299] Negotiation successful
0926 16:32:59.348378 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":396,"threads_started":1}
I20260926 16:32:59.355420 17628 raft_consensus.cc:348] T 00000000000000000000000000000000 P f34b2bc957b8474fa4bcb5ff02ffd06b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f34b2bc957b8474fa4bcb5ff02ffd06b" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 39201 } }
I20260926 16:32:59.355937 17628 raft_consensus.cc:374] T 00000000000000000000000000000000 P f34b2bc957b8474fa4bcb5ff02ffd06b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:32:59.356089 17628 raft_consensus.cc:729] T 00000000000000000000000000000000 P f34b2bc957b8474fa4bcb5ff02ffd06b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f34b2bc957b8474fa4bcb5ff02ffd06b, State: Initialized, Role: FOLLOWER
I20260926 16:32:59.356727 17628 consensus_queue.cc:260] T 00000000000000000000000000000000 P f34b2bc957b8474fa4bcb5ff02ffd06b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f34b2bc957b8474fa4bcb5ff02ffd06b" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 39201 } }
I20260926 16:32:59.357111 17628 raft_consensus.cc:388] T 00000000000000000000000000000000 P f34b2bc957b8474fa4bcb5ff02ffd06b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:32:59.357281 17628 raft_consensus.cc:482] T 00000000000000000000000000000000 P f34b2bc957b8474fa4bcb5ff02ffd06b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:32:59.357474 17628 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f34b2bc957b8474fa4bcb5ff02ffd06b [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:32:59.360394 17628 raft_consensus.cc:504] T 00000000000000000000000000000000 P f34b2bc957b8474fa4bcb5ff02ffd06b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f34b2bc957b8474fa4bcb5ff02ffd06b" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 39201 } }
I20260926 16:32:59.361191 17628 leader_election.cc:302] T 00000000000000000000000000000000 P f34b2bc957b8474fa4bcb5ff02ffd06b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f34b2bc957b8474fa4bcb5ff02ffd06b; no voters:
I20260926 16:32:59.361884 17628 leader_election.cc:288] T 00000000000000000000000000000000 P f34b2bc957b8474fa4bcb5ff02ffd06b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:32:59.362043 17633 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f34b2bc957b8474fa4bcb5ff02ffd06b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:32:59.362732 17633 raft_consensus.cc:686] T 00000000000000000000000000000000 P f34b2bc957b8474fa4bcb5ff02ffd06b [term 1 LEADER]: Becoming Leader. State: Replica: f34b2bc957b8474fa4bcb5ff02ffd06b, State: Running, Role: LEADER
I20260926 16:32:59.363299 17633 consensus_queue.cc:237] T 00000000000000000000000000000000 P f34b2bc957b8474fa4bcb5ff02ffd06b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f34b2bc957b8474fa4bcb5ff02ffd06b" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 39201 } }
I20260926 16:32:59.364259 17628 sys_catalog.cc:564] T 00000000000000000000000000000000 P f34b2bc957b8474fa4bcb5ff02ffd06b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:32:59.365988 17635 sys_catalog.cc:455] T 00000000000000000000000000000000 P f34b2bc957b8474fa4bcb5ff02ffd06b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f34b2bc957b8474fa4bcb5ff02ffd06b. Latest consensus state: current_term: 1 leader_uuid: "f34b2bc957b8474fa4bcb5ff02ffd06b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f34b2bc957b8474fa4bcb5ff02ffd06b" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 39201 } } }
I20260926 16:32:59.366420 17635 sys_catalog.cc:458] T 00000000000000000000000000000000 P f34b2bc957b8474fa4bcb5ff02ffd06b [sys.catalog]: This master's current role is: LEADER
I20260926 16:32:59.366138 17634 sys_catalog.cc:455] T 00000000000000000000000000000000 P f34b2bc957b8474fa4bcb5ff02ffd06b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f34b2bc957b8474fa4bcb5ff02ffd06b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f34b2bc957b8474fa4bcb5ff02ffd06b" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 39201 } } }
I20260926 16:32:59.366698 17634 sys_catalog.cc:458] T 00000000000000000000000000000000 P f34b2bc957b8474fa4bcb5ff02ffd06b [sys.catalog]: This master's current role is: LEADER
I20260926 16:32:59.374065 17638 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:32:59.381160 17638 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:32:59.382014 17628 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 16:32:59.400524 17638 catalog_manager.cc:1347] Generated new cluster ID: da07fa89c02148d8ab351a67e8dc7b38
I20260926 16:32:59.400718 17638 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:32:59.411413 17638 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:32:59.413583 17638 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:32:59.422883 17638 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f34b2bc957b8474fa4bcb5ff02ffd06b: Generated new TSK 0
I20260926 16:32:59.424081 17638 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.33.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab.
Entry for principal kudu/127.6.33.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab.
I20260926 16:33:00.169199 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.33.20:40679/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.1:0
--local_ip_for_outbound_sockets=127.6.33.1
--webserver_interface=127.6.33.1
--webserver_port=0
--tserver_master_addrs=127.6.33.62:39201
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab
--principal=kudu/127.6.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:43483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 16:33:00 dist-test-slave-khd3 krb5kdc[17117](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790440380, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 16:33:00.595777 17673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:33:00.596189 17673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:33:00.596400 17673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:33:00.608240 17673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:33:00.608467 17673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:33:00.608693 17673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.1
I20260926 16:33:00.624892 17673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:43483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.6.33.20:40679/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab
--principal=kudu/127.6.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.33.1
--webserver_port=0
--tserver_master_addrs=127.6.33.62:39201
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.1
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:33:00.626830 17673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:33:00.628594 17673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:33:00.646469 17683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:33:00.646492 17684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:33:00.658401 17686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:33:00.659049 17673 server_base.cc:1034] running on GCE node
I20260926 16:33:00.659798 17673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:33:00.660972 17673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:33:00.662231 17673 hybrid_clock.cc:648] HybridClock initialized: now 1790440380662153 us; error 69 us; skew 500 ppm
I20260926 16:33:00.662709 17673 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:33:00 dist-test-slave-khd3 krb5kdc[17117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440380, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:33:00.666181 17673 init.cc:377] Logged in from keytab as kudu/127.6.33.1@KRBTEST.COM (short username kudu)
I20260926 16:33:00.668243 17673 webserver.cc:460] Webserver started at http://127.6.33.1:39263/ using document root <none> and password file <none>
I20260926 16:33:00.669122 17673 fs_manager.cc:359] Metadata directory not provided
I20260926 16:33:00.669268 17673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:33:00.669610 17673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 16:33:00 dist-test-slave-khd3 krb5kdc[17117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440380, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.1@KRBTEST.COM for HTTP/127.6.33.20@KRBTEST.COM
I20260926 16:33:00.701021 17673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ts-0/data/instance:
uuid: "cb633f392ea64bdc8efc2d5e442c6f44"
format_stamp: "Formatted at 2026-09-26 16:33:00 on dist-test-slave-khd3"
server_key: "57c4654270bb89ce6355bc6e2e4d6732"
server_key_iv: "3fbaa09d20aaabac960b5e29236892ec"
server_key_version: "kuduclusterkey@0"
I20260926 16:33:00.702214 17673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ts-0/wal/instance:
uuid: "cb633f392ea64bdc8efc2d5e442c6f44"
format_stamp: "Formatted at 2026-09-26 16:33:00 on dist-test-slave-khd3"
server_key: "57c4654270bb89ce6355bc6e2e4d6732"
server_key_iv: "3fbaa09d20aaabac960b5e29236892ec"
server_key_version: "kuduclusterkey@0"
I20260926 16:33:00.709966 17673 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260926 16:33:00.731582 17693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:33:00.734117 17673 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260926 16:33:00.734323 17673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ts-0/wal
uuid: "cb633f392ea64bdc8efc2d5e442c6f44"
format_stamp: "Formatted at 2026-09-26 16:33:00 on dist-test-slave-khd3"
server_key: "57c4654270bb89ce6355bc6e2e4d6732"
server_key_iv: "3fbaa09d20aaabac960b5e29236892ec"
server_key_version: "kuduclusterkey@0"
I20260926 16:33:00.734606 17673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:33:00.760610 17673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:33:00.765790 17673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:33:00.766145 17673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:33:00.767331 17673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:33:00.769253 17673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:33:00.769361 17673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:33:00.769500 17673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:33:00.769582 17673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:33:00.826792 17673 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.1:34579
I20260926 16:33:00.826802 17805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.1:34579 every 8 connection(s)
I20260926 16:33:00.828289 17673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
I20260926 16:33:00.836336 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 17673
I20260926 16:33:00.836611 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ts-0/wal/instance
I20260926 16:33:00.837778 6276 external_mini_cluster.cc:1428] Setting key 7dee4f685a91a3e4497f964404674d18
Sep 26 16:33:00 dist-test-slave-khd3 krb5kdc[17117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440380, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.1@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.33.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.2@KRBTEST.COM" created.
I20260926 16:33:00.872469 17809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:33:00.832384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:39201 (local address 127.6.33.1:50773)
0926 16:33:00.842370 (+ 9986us) negotiation.cc:107] Waiting for socket to connect
0926 16:33:00.842430 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 16:33:00.845217 (+ 2787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:33:00.850023 (+ 4806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:33:00.850037 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:33:00.850678 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:33:00.853244 (+ 2566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:33:00.853277 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:33:00.854940 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:33:00.854946 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:33:00.857139 (+ 2193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:33:00.857156 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:33:00.858541 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:33:00.858550 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:33:00.858861 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:33:00.859747 (+ 886us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:33:00.859780 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:33:00.862709 (+ 2929us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:33:00.867111 (+ 4402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:33:00.867173 (+ 62us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:33:00.867221 (+ 48us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:33:00.867975 (+ 754us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:33:00.868587 (+ 612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:33:00.868592 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:33:00.868596 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:33:00.868873 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:33:00.869612 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:33:00.869620 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:33:00.870067 (+ 447us) client_negotiation.cc:769] Sending connection context
0926 16:33:00.870386 (+ 319us) client_negotiation.cc:240] Negotiation successful
0926 16:33:00.870965 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9648,"thread_start_us":15609,"threads_started":1}
I20260926 16:33:00.874424 17808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:33:00.831812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:50773 (local address 127.6.33.62:39201)
0926 16:33:00.849192 (+ 17380us) server_negotiation.cc:206] Beginning negotiation
0926 16:33:00.849199 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:33:00.849222 (+ 23us) server_negotiation.cc:407] Connection header received
0926 16:33:00.849288 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:33:00.849297 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:33:00.849391 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:33:00.849516 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:33:00.853545 (+ 4029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:00.854710 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:00.857342 (+ 2632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:00.858056 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:00.858189 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:33:00.862997 (+ 4808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:33:00.863004 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:33:00.863009 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:33:00.863054 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:33:00.866928 (+ 3874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:33:00.868190 (+ 1262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:33:00.868196 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:33:00.868200 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:33:00.868394 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:33:00.869013 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:33:00.869018 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:33:00.869022 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:33:00.869371 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:33:00.869529 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 16:33:00.873778 (+ 4249us) server_negotiation.cc:299] Negotiation successful
0926 16:33:00.874029 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17240,"thread_start_us":17113,"threads_started":1}
I20260926 16:33:00.876209 17806 heartbeater.cc:347] Connected to a master server at 127.6.33.62:39201
I20260926 16:33:00.876573 17806 heartbeater.cc:464] Registering TS with master...
I20260926 16:33:00.877599 17806 heartbeater.cc:511] Master 127.6.33.62:39201 requested a full tablet report, sending...
I20260926 16:33:00.880362 17593 ts_manager.cc:194] Registered new tserver with Master: cb633f392ea64bdc8efc2d5e442c6f44 (127.6.33.1:34579)
I20260926 16:33:00.882614 17593 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.33.1@KRBTEST.COM'} at 127.6.33.1:50773
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab.
Entry for principal kudu/127.6.33.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab.
I20260926 16:33:00.908694 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.33.20:40679/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.2:0
--local_ip_for_outbound_sockets=127.6.33.2
--webserver_interface=127.6.33.2
--webserver_port=0
--tserver_master_addrs=127.6.33.62:39201
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab
--principal=kudu/127.6.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:43483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 16:33:00 dist-test-slave-khd3 krb5kdc[17117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440380, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.62@KRBTEST.COM for HTTP/127.6.33.20@KRBTEST.COM
Sep 26 16:33:01 dist-test-slave-khd3 krb5kdc[17117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440380, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.62@KRBTEST.COM for HTTP/127.6.33.20@KRBTEST.COM
W20260926 16:33:01.435045 17812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:33:01.435581 17812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:33:01.435760 17812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:33:01.446926 17812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:33:01.447216 17812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:33:01.447435 17812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.2
I20260926 16:33:01.462780 17812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:43483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.6.33.20:40679/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab
--principal=kudu/127.6.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.33.2
--webserver_port=0
--tserver_master_addrs=127.6.33.62:39201
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.2
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:33:01.464895 17812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:33:01.466650 17812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:33:01.506860 17820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:33:01.514806 17823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:33:01.515904 17819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:33:01.516551 17812 server_base.cc:1034] running on GCE node
I20260926 16:33:01.517623 17812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:33:01.523880 17812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:33:01.525417 17812 hybrid_clock.cc:648] HybridClock initialized: now 1790440381525300 us; error 117 us; skew 500 ppm
I20260926 16:33:01.525899 17812 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:33:01 dist-test-slave-khd3 krb5kdc[17117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440381, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:33:01.536544 17812 init.cc:377] Logged in from keytab as kudu/127.6.33.2@KRBTEST.COM (short username kudu)
I20260926 16:33:01.568950 17812 webserver.cc:460] Webserver started at http://127.6.33.2:41649/ using document root <none> and password file <none>
I20260926 16:33:01.570366 17812 fs_manager.cc:359] Metadata directory not provided
I20260926 16:33:01.570623 17812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:33:01.571069 17812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 16:33:01 dist-test-slave-khd3 krb5kdc[17117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440381, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.2@KRBTEST.COM for HTTP/127.6.33.20@KRBTEST.COM
I20260926 16:33:01.604874 17812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ts-1/data/instance:
uuid: "21eac8d83b5b4a2bb11061763bd82b6f"
format_stamp: "Formatted at 2026-09-26 16:33:01 on dist-test-slave-khd3"
server_key: "3e36213e21fbc90ccbe7db0788746132"
server_key_iv: "4ad2e637ef5ce5fa892d54c572462043"
server_key_version: "kuduclusterkey@0"
I20260926 16:33:01.607235 17812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ts-1/wal/instance:
uuid: "21eac8d83b5b4a2bb11061763bd82b6f"
format_stamp: "Formatted at 2026-09-26 16:33:01 on dist-test-slave-khd3"
server_key: "3e36213e21fbc90ccbe7db0788746132"
server_key_iv: "4ad2e637ef5ce5fa892d54c572462043"
server_key_version: "kuduclusterkey@0"
I20260926 16:33:01.621495 17812 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.008s sys 0.004s
I20260926 16:33:01.676254 17836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:33:01.678804 17812 fs_manager.cc:725] Time spent opening block manager: real 0.020s user 0.003s sys 0.001s
I20260926 16:33:01.679127 17812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ts-1/wal
uuid: "21eac8d83b5b4a2bb11061763bd82b6f"
format_stamp: "Formatted at 2026-09-26 16:33:01 on dist-test-slave-khd3"
server_key: "3e36213e21fbc90ccbe7db0788746132"
server_key_iv: "4ad2e637ef5ce5fa892d54c572462043"
server_key_version: "kuduclusterkey@0"
I20260926 16:33:01.679483 17812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:33:01.774646 17812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:33:01.779155 17812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:33:01.779511 17812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:33:01.780578 17812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:33:01.782181 17812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:33:01.782282 17812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:33:01.782415 17812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:33:01.782490 17812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:33:01.824095 17812 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.2:38071
I20260926 16:33:01.824115 17948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.2:38071 every 8 connection(s)
I20260926 16:33:01.825464 17812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
I20260926 16:33:01.827492 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 17812
I20260926 16:33:01.827715 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ts-1/wal/instance
I20260926 16:33:01.828356 6276 external_mini_cluster.cc:1428] Setting key 141c0b140bd1e326e1cdf12da25e4b18
Sep 26 16:33:01 dist-test-slave-khd3 krb5kdc[17117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440381, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.2@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.33.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.3@KRBTEST.COM" created.
I20260926 16:33:01.859159 17952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:33:01.831709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:50735 (local address 127.6.33.62:39201)
0926 16:33:01.839644 (+ 7935us) server_negotiation.cc:206] Beginning negotiation
0926 16:33:01.839652 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:33:01.839674 (+ 22us) server_negotiation.cc:407] Connection header received
0926 16:33:01.839744 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:33:01.839751 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:33:01.839845 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:33:01.840443 (+ 598us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:33:01.842114 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:01.843358 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:01.845999 (+ 2641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:01.846721 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:01.846858 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:33:01.851649 (+ 4791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:33:01.851658 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:33:01.851663 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:33:01.851709 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:33:01.855656 (+ 3947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:33:01.856679 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:33:01.856683 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:33:01.856685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:33:01.856869 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:33:01.857306 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:33:01.857311 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:33:01.857313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:33:01.857598 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:33:01.857742 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 16:33:01.858544 (+ 802us) server_negotiation.cc:299] Negotiation successful
0926 16:33:01.858773 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7775,"thread_start_us":7661,"threads_started":1}
I20260926 16:33:01.860263 17951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:33:01.830993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:39201 (local address 127.6.33.2:50735)
0926 16:33:01.833684 (+ 2691us) negotiation.cc:107] Waiting for socket to connect
0926 16:33:01.833741 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 16:33:01.836466 (+ 2725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:33:01.840301 (+ 3835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:33:01.840316 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:33:01.841045 (+ 729us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:33:01.841911 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:33:01.841943 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:33:01.843553 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:33:01.843559 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:33:01.845832 (+ 2273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:33:01.845848 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:33:01.846974 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:33:01.846984 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:33:01.847300 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:33:01.848164 (+ 864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:33:01.848197 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:33:01.851357 (+ 3160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:33:01.855812 (+ 4455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:33:01.855820 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:33:01.855834 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:33:01.856485 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:33:01.856982 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:33:01.856987 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:33:01.856991 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:33:01.857185 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:33:01.857734 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:33:01.857742 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:33:01.858149 (+ 407us) client_negotiation.cc:769] Sending connection context
0926 16:33:01.858430 (+ 281us) client_negotiation.cc:240] Negotiation successful
0926 16:33:01.858958 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2365,"thread_start_us":2110,"threads_started":1}
I20260926 16:33:01.861847 17949 heartbeater.cc:347] Connected to a master server at 127.6.33.62:39201
I20260926 16:33:01.862167 17949 heartbeater.cc:464] Registering TS with master...
I20260926 16:33:01.862916 17949 heartbeater.cc:511] Master 127.6.33.62:39201 requested a full tablet report, sending...
I20260926 16:33:01.864410 17593 ts_manager.cc:194] Registered new tserver with Master: 21eac8d83b5b4a2bb11061763bd82b6f (127.6.33.2:38071)
I20260926 16:33:01.865365 17593 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.33.2@KRBTEST.COM'} at 127.6.33.2:50735
I20260926 16:33:01.886899 17806 heartbeater.cc:503] Master 127.6.33.62:39201 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab.
Entry for principal kudu/127.6.33.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab.
I20260926 16:33:01.892518 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.33.20:40679/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.3:0
--local_ip_for_outbound_sockets=127.6.33.3
--webserver_interface=127.6.33.3
--webserver_port=0
--tserver_master_addrs=127.6.33.62:39201
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab
--principal=kudu/127.6.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:43483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:33:02.271914 17955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:33:02.272246 17955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:33:02.272418 17955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:33:02.280037 17955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:33:02.280208 17955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:33:02.280403 17955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.3
I20260926 16:33:02.290722 17955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:43483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.6.33.20:40679/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab
--principal=kudu/127.6.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.33.3
--webserver_port=0
--tserver_master_addrs=127.6.33.62:39201
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.3
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:33:02.292068 17955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:33:02.293479 17955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:33:02.307375 17960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:33:02.307423 17961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:33:02.307664 17963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:33:02.308102 17955 server_base.cc:1034] running on GCE node
I20260926 16:33:02.308799 17955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:33:02.309845 17955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:33:02.311115 17955 hybrid_clock.cc:648] HybridClock initialized: now 1790440382311051 us; error 59 us; skew 500 ppm
I20260926 16:33:02.311477 17955 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:33:02 dist-test-slave-khd3 krb5kdc[17117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440382, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:33:02.314971 17955 init.cc:377] Logged in from keytab as kudu/127.6.33.3@KRBTEST.COM (short username kudu)
I20260926 16:33:02.316608 17955 webserver.cc:460] Webserver started at http://127.6.33.3:45625/ using document root <none> and password file <none>
I20260926 16:33:02.317412 17955 fs_manager.cc:359] Metadata directory not provided
I20260926 16:33:02.317567 17955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:33:02.317924 17955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 16:33:02 dist-test-slave-khd3 krb5kdc[17117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440382, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.3@KRBTEST.COM for HTTP/127.6.33.20@KRBTEST.COM
I20260926 16:33:02.336064 17955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ts-2/data/instance:
uuid: "afa5537dc2ff4fce979f4664bc607ee5"
format_stamp: "Formatted at 2026-09-26 16:33:02 on dist-test-slave-khd3"
server_key: "a95d7e41b028d07009acfdb284c7230f"
server_key_iv: "8aad50f08857d892ad280607c99b5d98"
server_key_version: "kuduclusterkey@0"
I20260926 16:33:02.337430 17955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ts-2/wal/instance:
uuid: "afa5537dc2ff4fce979f4664bc607ee5"
format_stamp: "Formatted at 2026-09-26 16:33:02 on dist-test-slave-khd3"
server_key: "a95d7e41b028d07009acfdb284c7230f"
server_key_iv: "8aad50f08857d892ad280607c99b5d98"
server_key_version: "kuduclusterkey@0"
I20260926 16:33:02.344305 17955 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 16:33:02.363224 17970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:33:02.365094 17955 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260926 16:33:02.365298 17955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ts-2/wal
uuid: "afa5537dc2ff4fce979f4664bc607ee5"
format_stamp: "Formatted at 2026-09-26 16:33:02 on dist-test-slave-khd3"
server_key: "a95d7e41b028d07009acfdb284c7230f"
server_key_iv: "8aad50f08857d892ad280607c99b5d98"
server_key_version: "kuduclusterkey@0"
I20260926 16:33:02.365551 17955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:33:02.385857 17955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:33:02.389871 17955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:33:02.390177 17955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:33:02.391189 17955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:33:02.392565 17955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:33:02.392639 17955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:33:02.392823 17955 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:33:02.392874 17955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260926 16:33:02.426085 17955 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.3:36785
I20260926 16:33:02.426121 18082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.3:36785 every 8 connection(s)
I20260926 16:33:02.427373 17955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
I20260926 16:33:02.432111 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 17955
I20260926 16:33:02.432313 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ts-2/wal/instance
I20260926 16:33:02.432794 6276 external_mini_cluster.cc:1428] Setting key 8377546b9a02fa5a2386d798aeed0925
Sep 26 16:33:02 dist-test-slave-khd3 krb5kdc[17117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440382, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.3@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:33:02.449141 18085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:33:02.430039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.3:34205 (local address 127.6.33.62:39201)
0926 16:33:02.430536 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0926 16:33:02.430541 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:33:02.432469 (+ 1928us) server_negotiation.cc:407] Connection header received
0926 16:33:02.433886 (+ 1417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:33:02.433891 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:33:02.434005 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:33:02.434118 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:33:02.435551 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:02.436265 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:02.438017 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:02.438473 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:02.438614 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:33:02.442479 (+ 3865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:33:02.442484 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:33:02.442488 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:33:02.442528 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:33:02.445461 (+ 2933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:33:02.446295 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:33:02.446300 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:33:02.446303 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:33:02.446504 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:33:02.447137 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:33:02.447141 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:33:02.447143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:33:02.447417 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:33:02.447586 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 16:33:02.448492 (+ 906us) server_negotiation.cc:299] Negotiation successful
0926 16:33:02.448733 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":225,"threads_started":1}
I20260926 16:33:02.450147 18086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:33:02.430656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:39201 (local address 127.6.33.3:34205)
0926 16:33:02.431545 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0926 16:33:02.431632 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0926 16:33:02.433623 (+ 1991us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:33:02.434360 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:33:02.434368 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:33:02.434791 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:33:02.435392 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:33:02.435408 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:33:02.436435 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:33:02.436447 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:33:02.437837 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:33:02.437849 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:33:02.438720 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:33:02.438726 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:33:02.438973 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:33:02.439681 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:33:02.439705 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:33:02.442216 (+ 2511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:33:02.445640 (+ 3424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:33:02.445650 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:33:02.445673 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:33:02.446140 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:33:02.446616 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:33:02.446619 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:33:02.446622 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:33:02.446944 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:33:02.447586 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:33:02.447597 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:33:02.448045 (+ 448us) client_negotiation.cc:769] Sending connection context
0926 16:33:02.448396 (+ 351us) client_negotiation.cc:240] Negotiation successful
0926 16:33:02.449002 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":355,"threads_started":1}
I20260926 16:33:02.451627 18083 heartbeater.cc:347] Connected to a master server at 127.6.33.62:39201
I20260926 16:33:02.451916 18083 heartbeater.cc:464] Registering TS with master...
I20260926 16:33:02.452644 18083 heartbeater.cc:511] Master 127.6.33.62:39201 requested a full tablet report, sending...
I20260926 16:33:02.454185 17593 ts_manager.cc:194] Registered new tserver with Master: afa5537dc2ff4fce979f4664bc607ee5 (127.6.33.3:36785)
I20260926 16:33:02.455111 17593 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.33.3@KRBTEST.COM'} at 127.6.33.3:34205
I20260926 16:33:02.461305 6276 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:33:02.480216 18085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:33:02.466320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43886 (local address 127.6.33.62:39201)
0926 16:33:02.466508 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 16:33:02.466512 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:33:02.467002 (+ 490us) server_negotiation.cc:407] Connection header received
0926 16:33:02.467189 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:33:02.467202 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:33:02.467291 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:33:02.467387 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:33:02.469089 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:02.469855 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:02.471378 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:02.471834 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:02.471934 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:33:02.474409 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:33:02.474414 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:33:02.474417 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:33:02.474449 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:33:02.477611 (+ 3162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:33:02.478335 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:33:02.478339 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:33:02.478341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:33:02.478468 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:33:02.478955 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:33:02.478959 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:33:02.478961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:33:02.479282 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:33:02.479477 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0926 16:33:02.479752 (+ 275us) server_negotiation.cc:299] Negotiation successful
0926 16:33:02.479933 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 16:33:02.484663 17593 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43886:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:33:02.491892 17593 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:33:02.518301 18096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:33:02.504991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.2:38071 (local address 127.0.0.1:33784)
0926 16:33:02.505889 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0926 16:33:02.505939 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 16:33:02.506089 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:33:02.507539 (+ 1450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:33:02.507550 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:33:02.507585 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:33:02.508114 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:33:02.508130 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:33:02.510729 (+ 2599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:33:02.510739 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:33:02.514001 (+ 3262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:33:02.514023 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:33:02.516936 (+ 2913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:33:02.516945 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:33:02.517249 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:33:02.517322 (+ 73us) client_negotiation.cc:769] Sending connection context
0926 16:33:02.517479 (+ 157us) client_negotiation.cc:240] Negotiation successful
0926 16:33:02.517740 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":749,"threads_started":1}
I20260926 16:33:02.519851 18097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:33:02.505261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33784 (local address 127.6.33.2:38071)
0926 16:33:02.505834 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0926 16:33:02.505845 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:33:02.506041 (+ 196us) server_negotiation.cc:407] Connection header received
0926 16:33:02.506270 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:33:02.506278 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:33:02.506545 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:33:02.506769 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:33:02.508370 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:02.510506 (+ 2136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:02.514258 (+ 3752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:02.516356 (+ 2098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:02.516526 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:33:02.518942 (+ 2416us) server_negotiation.cc:1035] Waiting for connection context
0926 16:33:02.519052 (+ 110us) server_negotiation.cc:299] Negotiation successful
0926 16:33:02.519294 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":268,"threads_started":1}
I20260926 16:33:02.525842 17884 tablet_service.cc:1467] Processing CreateTablet for tablet 39e70c9f15f44c519b6e7def43b2dd3d (DEFAULT_TABLE table=test-table [id=e2e86caa01ea4699901db0ef75b3861b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:33:02.527483 17884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39e70c9f15f44c519b6e7def43b2dd3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:33:02.527947 18098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:33:02.506553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.3:36785 (local address 127.0.0.1:59916)
0926 16:33:02.507184 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0926 16:33:02.507204 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 16:33:02.507360 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:33:02.521286 (+ 13926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:33:02.521291 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:33:02.521319 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:33:02.521761 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:33:02.521774 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:33:02.523559 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:33:02.523566 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:33:02.525161 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:33:02.525175 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:33:02.526952 (+ 1777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:33:02.526958 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:33:02.527140 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:33:02.527178 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 16:33:02.527276 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 16:33:02.527419 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":263,"threads_started":1}
I20260926 16:33:02.528510 18096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:33:02.507206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:34579 (local address 127.0.0.1:46288)
0926 16:33:02.518584 (+ 11378us) negotiation.cc:107] Waiting for socket to connect
0926 16:33:02.518601 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 16:33:02.518710 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:33:02.521735 (+ 3025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:33:02.521744 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:33:02.521769 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:33:02.522662 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:33:02.522673 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:33:02.524522 (+ 1849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:33:02.524528 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:33:02.525962 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:33:02.525979 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:33:02.527788 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:33:02.527793 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:33:02.528002 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:33:02.528045 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 16:33:02.528135 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 16:33:02.528270 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11259,"thread_start_us":14419,"threads_started":1}
I20260926 16:33:02.529053 18099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:33:02.506921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59916 (local address 127.6.33.3:36785)
0926 16:33:02.520496 (+ 13575us) server_negotiation.cc:206] Beginning negotiation
0926 16:33:02.520507 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:33:02.520544 (+ 37us) server_negotiation.cc:407] Connection header received
0926 16:33:02.520695 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:33:02.520704 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:33:02.521049 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:33:02.521260 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:33:02.522239 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:02.523424 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:02.525333 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:02.526734 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:02.526859 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:33:02.527108 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0926 16:33:02.528502 (+ 1394us) server_negotiation.cc:299] Negotiation successful
0926 16:33:02.528673 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13399,"thread_start_us":13241,"threads_started":1}
I20260926 16:33:02.530136 18101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:33:02.507545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46288 (local address 127.6.33.1:34579)
0926 16:33:02.520929 (+ 13384us) server_negotiation.cc:206] Beginning negotiation
0926 16:33:02.520938 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:33:02.520964 (+ 26us) server_negotiation.cc:407] Connection header received
0926 16:33:02.521057 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:33:02.521064 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:33:02.521288 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:33:02.522421 (+ 1133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:33:02.522806 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:02.524360 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:02.526124 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:02.527361 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:02.527470 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:33:02.527702 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0926 16:33:02.528747 (+ 1045us) server_negotiation.cc:299] Negotiation successful
0926 16:33:02.528959 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13176,"thread_start_us":13032,"threads_started":1}
I20260926 16:33:02.532348 18018 tablet_service.cc:1467] Processing CreateTablet for tablet 39e70c9f15f44c519b6e7def43b2dd3d (DEFAULT_TABLE table=test-table [id=e2e86caa01ea4699901db0ef75b3861b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:33:02.533646 18018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39e70c9f15f44c519b6e7def43b2dd3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:33:02.533771 17741 tablet_service.cc:1467] Processing CreateTablet for tablet 39e70c9f15f44c519b6e7def43b2dd3d (DEFAULT_TABLE table=test-table [id=e2e86caa01ea4699901db0ef75b3861b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:33:02.535198 17741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39e70c9f15f44c519b6e7def43b2dd3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:33:02.542171 18102 tablet_bootstrap.cc:492] T 39e70c9f15f44c519b6e7def43b2dd3d P 21eac8d83b5b4a2bb11061763bd82b6f: Bootstrap starting.
I20260926 16:33:02.546346 18103 tablet_bootstrap.cc:492] T 39e70c9f15f44c519b6e7def43b2dd3d P afa5537dc2ff4fce979f4664bc607ee5: Bootstrap starting.
I20260926 16:33:02.546433 18102 tablet_bootstrap.cc:654] T 39e70c9f15f44c519b6e7def43b2dd3d P 21eac8d83b5b4a2bb11061763bd82b6f: Neither blocks nor log segments found. Creating new log.
I20260926 16:33:02.547905 18102 log.cc:824] T 39e70c9f15f44c519b6e7def43b2dd3d P 21eac8d83b5b4a2bb11061763bd82b6f: Log is configured to *not* fsync() on all Append() calls
I20260926 16:33:02.550230 18103 tablet_bootstrap.cc:654] T 39e70c9f15f44c519b6e7def43b2dd3d P afa5537dc2ff4fce979f4664bc607ee5: Neither blocks nor log segments found. Creating new log.
I20260926 16:33:02.550681 18105 tablet_bootstrap.cc:492] T 39e70c9f15f44c519b6e7def43b2dd3d P cb633f392ea64bdc8efc2d5e442c6f44: Bootstrap starting.
I20260926 16:33:02.551481 18103 log.cc:824] T 39e70c9f15f44c519b6e7def43b2dd3d P afa5537dc2ff4fce979f4664bc607ee5: Log is configured to *not* fsync() on all Append() calls
I20260926 16:33:02.551604 18102 tablet_bootstrap.cc:492] T 39e70c9f15f44c519b6e7def43b2dd3d P 21eac8d83b5b4a2bb11061763bd82b6f: No bootstrap required, opened a new log
I20260926 16:33:02.551904 18102 ts_tablet_manager.cc:1397] T 39e70c9f15f44c519b6e7def43b2dd3d P 21eac8d83b5b4a2bb11061763bd82b6f: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260926 16:33:02.554404 18103 tablet_bootstrap.cc:492] T 39e70c9f15f44c519b6e7def43b2dd3d P afa5537dc2ff4fce979f4664bc607ee5: No bootstrap required, opened a new log
I20260926 16:33:02.554561 18105 tablet_bootstrap.cc:654] T 39e70c9f15f44c519b6e7def43b2dd3d P cb633f392ea64bdc8efc2d5e442c6f44: Neither blocks nor log segments found. Creating new log.
I20260926 16:33:02.554698 18103 ts_tablet_manager.cc:1397] T 39e70c9f15f44c519b6e7def43b2dd3d P afa5537dc2ff4fce979f4664bc607ee5: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 16:33:02.555832 18105 log.cc:824] T 39e70c9f15f44c519b6e7def43b2dd3d P cb633f392ea64bdc8efc2d5e442c6f44: Log is configured to *not* fsync() on all Append() calls
I20260926 16:33:02.559281 18105 tablet_bootstrap.cc:492] T 39e70c9f15f44c519b6e7def43b2dd3d P cb633f392ea64bdc8efc2d5e442c6f44: No bootstrap required, opened a new log
I20260926 16:33:02.559578 18105 ts_tablet_manager.cc:1397] T 39e70c9f15f44c519b6e7def43b2dd3d P cb633f392ea64bdc8efc2d5e442c6f44: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 16:33:02.562911 18102 raft_consensus.cc:348] T 39e70c9f15f44c519b6e7def43b2dd3d P 21eac8d83b5b4a2bb11061763bd82b6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eac8d83b5b4a2bb11061763bd82b6f" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 38071 } } peers { permanent_uuid: "afa5537dc2ff4fce979f4664bc607ee5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 36785 } } peers { permanent_uuid: "cb633f392ea64bdc8efc2d5e442c6f44" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 34579 } }
I20260926 16:33:02.563427 18102 raft_consensus.cc:374] T 39e70c9f15f44c519b6e7def43b2dd3d P 21eac8d83b5b4a2bb11061763bd82b6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:33:02.563544 18102 raft_consensus.cc:729] T 39e70c9f15f44c519b6e7def43b2dd3d P 21eac8d83b5b4a2bb11061763bd82b6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21eac8d83b5b4a2bb11061763bd82b6f, State: Initialized, Role: FOLLOWER
I20260926 16:33:02.563194 18103 raft_consensus.cc:348] T 39e70c9f15f44c519b6e7def43b2dd3d P afa5537dc2ff4fce979f4664bc607ee5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eac8d83b5b4a2bb11061763bd82b6f" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 38071 } } peers { permanent_uuid: "afa5537dc2ff4fce979f4664bc607ee5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 36785 } } peers { permanent_uuid: "cb633f392ea64bdc8efc2d5e442c6f44" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 34579 } }
I20260926 16:33:02.563727 18103 raft_consensus.cc:374] T 39e70c9f15f44c519b6e7def43b2dd3d P afa5537dc2ff4fce979f4664bc607ee5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:33:02.563850 18103 raft_consensus.cc:729] T 39e70c9f15f44c519b6e7def43b2dd3d P afa5537dc2ff4fce979f4664bc607ee5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afa5537dc2ff4fce979f4664bc607ee5, State: Initialized, Role: FOLLOWER
I20260926 16:33:02.564407 18102 consensus_queue.cc:260] T 39e70c9f15f44c519b6e7def43b2dd3d P 21eac8d83b5b4a2bb11061763bd82b6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eac8d83b5b4a2bb11061763bd82b6f" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 38071 } } peers { permanent_uuid: "afa5537dc2ff4fce979f4664bc607ee5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 36785 } } peers { permanent_uuid: "cb633f392ea64bdc8efc2d5e442c6f44" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 34579 } }
I20260926 16:33:02.564639 18103 consensus_queue.cc:260] T 39e70c9f15f44c519b6e7def43b2dd3d P afa5537dc2ff4fce979f4664bc607ee5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eac8d83b5b4a2bb11061763bd82b6f" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 38071 } } peers { permanent_uuid: "afa5537dc2ff4fce979f4664bc607ee5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 36785 } } peers { permanent_uuid: "cb633f392ea64bdc8efc2d5e442c6f44" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 34579 } }
I20260926 16:33:02.565982 18083 heartbeater.cc:503] Master 127.6.33.62:39201 was elected leader, sending a full tablet report...
I20260926 16:33:02.566708 18103 ts_tablet_manager.cc:1428] T 39e70c9f15f44c519b6e7def43b2dd3d P afa5537dc2ff4fce979f4664bc607ee5: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260926 16:33:02.571800 17949 heartbeater.cc:503] Master 127.6.33.62:39201 was elected leader, sending a full tablet report...
I20260926 16:33:02.572950 18102 ts_tablet_manager.cc:1428] T 39e70c9f15f44c519b6e7def43b2dd3d P 21eac8d83b5b4a2bb11061763bd82b6f: Time spent starting tablet: real 0.021s user 0.016s sys 0.005s
I20260926 16:33:02.572868 18105 raft_consensus.cc:348] T 39e70c9f15f44c519b6e7def43b2dd3d P cb633f392ea64bdc8efc2d5e442c6f44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eac8d83b5b4a2bb11061763bd82b6f" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 38071 } } peers { permanent_uuid: "afa5537dc2ff4fce979f4664bc607ee5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 36785 } } peers { permanent_uuid: "cb633f392ea64bdc8efc2d5e442c6f44" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 34579 } }
I20260926 16:33:02.573401 18105 raft_consensus.cc:374] T 39e70c9f15f44c519b6e7def43b2dd3d P cb633f392ea64bdc8efc2d5e442c6f44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:33:02.573532 18105 raft_consensus.cc:729] T 39e70c9f15f44c519b6e7def43b2dd3d P cb633f392ea64bdc8efc2d5e442c6f44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb633f392ea64bdc8efc2d5e442c6f44, State: Initialized, Role: FOLLOWER
I20260926 16:33:02.574183 18105 consensus_queue.cc:260] T 39e70c9f15f44c519b6e7def43b2dd3d P cb633f392ea64bdc8efc2d5e442c6f44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eac8d83b5b4a2bb11061763bd82b6f" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 38071 } } peers { permanent_uuid: "afa5537dc2ff4fce979f4664bc607ee5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 36785 } } peers { permanent_uuid: "cb633f392ea64bdc8efc2d5e442c6f44" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 34579 } }
I20260926 16:33:02.575927 18105 ts_tablet_manager.cc:1428] T 39e70c9f15f44c519b6e7def43b2dd3d P cb633f392ea64bdc8efc2d5e442c6f44: Time spent starting tablet: real 0.016s user 0.017s sys 0.000s
W20260926 16:33:02.583438 17950 tablet.cc:2369] T 39e70c9f15f44c519b6e7def43b2dd3d P 21eac8d83b5b4a2bb11061763bd82b6f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 16:33:02.595141 17807 tablet.cc:2369] T 39e70c9f15f44c519b6e7def43b2dd3d P cb633f392ea64bdc8efc2d5e442c6f44: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 16:33:02.678987 18084 tablet.cc:2369] T 39e70c9f15f44c519b6e7def43b2dd3d P afa5537dc2ff4fce979f4664bc607ee5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:33:02.728116 18108 raft_consensus.cc:482] T 39e70c9f15f44c519b6e7def43b2dd3d P 21eac8d83b5b4a2bb11061763bd82b6f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:33:02.728487 18108 raft_consensus.cc:504] T 39e70c9f15f44c519b6e7def43b2dd3d P 21eac8d83b5b4a2bb11061763bd82b6f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eac8d83b5b4a2bb11061763bd82b6f" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 38071 } } peers { permanent_uuid: "afa5537dc2ff4fce979f4664bc607ee5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 36785 } } peers { permanent_uuid: "cb633f392ea64bdc8efc2d5e442c6f44" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 34579 } }
I20260926 16:33:02.730187 18108 leader_election.cc:288] T 39e70c9f15f44c519b6e7def43b2dd3d P 21eac8d83b5b4a2bb11061763bd82b6f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers afa5537dc2ff4fce979f4664bc607ee5 (127.6.33.3:36785), cb633f392ea64bdc8efc2d5e442c6f44 (127.6.33.1:34579)
I20260926 16:33:02.736919 18099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:33:02.730575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:46845 (local address 127.6.33.3:36785)
0926 16:33:02.730767 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 16:33:02.730772 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:33:02.731408 (+ 636us) server_negotiation.cc:407] Connection header received
0926 16:33:02.731665 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:33:02.731671 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:33:02.731759 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:33:02.731811 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:33:02.732579 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:02.733340 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:02.735215 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:02.735965 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:02.736116 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:33:02.736227 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 16:33:02.736514 (+ 287us) server_negotiation.cc:299] Negotiation successful
0926 16:33:02.736639 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 16:33:02.737008 18111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:33:02.730621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.3:36785 (local address 127.6.33.2:46845)
0926 16:33:02.731208 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0926 16:33:02.731262 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 16:33:02.731442 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:33:02.732014 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:33:02.732018 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:33:02.732045 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:33:02.732415 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:33:02.732429 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:33:02.733466 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:33:02.733474 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:33:02.735061 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:33:02.735072 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:33:02.736188 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:33:02.736193 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:33:02.736347 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:33:02.736373 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 16:33:02.736489 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 16:33:02.736629 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":328,"threads_started":1}
I20260926 16:33:02.737118 18101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:33:02.731497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:38383 (local address 127.6.33.1:34579)
0926 16:33:02.731691 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 16:33:02.731697 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:33:02.731946 (+ 249us) server_negotiation.cc:407] Connection header received
0926 16:33:02.732112 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:33:02.732143 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:33:02.732223 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:33:02.732317 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:33:02.732954 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:02.733661 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:02.735373 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:02.736149 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:02.736300 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:33:02.736403 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 16:33:02.736687 (+ 284us) server_negotiation.cc:299] Negotiation successful
0926 16:33:02.736850 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 16:33:02.737210 18112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:33:02.731282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:34579 (local address 127.6.33.2:38383)
0926 16:33:02.731841 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0926 16:33:02.731855 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 16:33:02.731956 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:33:02.732363 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:33:02.732370 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:33:02.732403 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:33:02.732741 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:33:02.732789 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:33:02.733855 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:33:02.733859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:33:02.735221 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:33:02.735232 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:33:02.736372 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:33:02.736377 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:33:02.736534 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:33:02.736556 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 16:33:02.736661 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 16:33:02.736807 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":314,"threads_started":1}
I20260926 16:33:02.738165 18038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e70c9f15f44c519b6e7def43b2dd3d" candidate_uuid: "21eac8d83b5b4a2bb11061763bd82b6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afa5537dc2ff4fce979f4664bc607ee5" is_pre_election: true
I20260926 16:33:02.738440 17761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e70c9f15f44c519b6e7def43b2dd3d" candidate_uuid: "21eac8d83b5b4a2bb11061763bd82b6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb633f392ea64bdc8efc2d5e442c6f44" is_pre_election: true
I20260926 16:33:02.738682 18038 raft_consensus.cc:2454] T 39e70c9f15f44c519b6e7def43b2dd3d P afa5537dc2ff4fce979f4664bc607ee5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21eac8d83b5b4a2bb11061763bd82b6f in term 0.
I20260926 16:33:02.738953 17761 raft_consensus.cc:2454] T 39e70c9f15f44c519b6e7def43b2dd3d P cb633f392ea64bdc8efc2d5e442c6f44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21eac8d83b5b4a2bb11061763bd82b6f in term 0.
I20260926 16:33:02.739432 17840 leader_election.cc:302] T 39e70c9f15f44c519b6e7def43b2dd3d P 21eac8d83b5b4a2bb11061763bd82b6f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21eac8d83b5b4a2bb11061763bd82b6f, afa5537dc2ff4fce979f4664bc607ee5; no voters:
I20260926 16:33:02.739843 18108 raft_consensus.cc:2781] T 39e70c9f15f44c519b6e7def43b2dd3d P 21eac8d83b5b4a2bb11061763bd82b6f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:33:02.739985 18108 raft_consensus.cc:482] T 39e70c9f15f44c519b6e7def43b2dd3d P 21eac8d83b5b4a2bb11061763bd82b6f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:33:02.740092 18108 raft_consensus.cc:3037] T 39e70c9f15f44c519b6e7def43b2dd3d P 21eac8d83b5b4a2bb11061763bd82b6f [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:33:02.742597 18108 raft_consensus.cc:504] T 39e70c9f15f44c519b6e7def43b2dd3d P 21eac8d83b5b4a2bb11061763bd82b6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eac8d83b5b4a2bb11061763bd82b6f" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 38071 } } peers { permanent_uuid: "afa5537dc2ff4fce979f4664bc607ee5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 36785 } } peers { permanent_uuid: "cb633f392ea64bdc8efc2d5e442c6f44" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 34579 } }
I20260926 16:33:02.743243 18108 leader_election.cc:288] T 39e70c9f15f44c519b6e7def43b2dd3d P 21eac8d83b5b4a2bb11061763bd82b6f [CANDIDATE]: Term 1 election: Requested vote from peers afa5537dc2ff4fce979f4664bc607ee5 (127.6.33.3:36785), cb633f392ea64bdc8efc2d5e442c6f44 (127.6.33.1:34579)
I20260926 16:33:02.743726 18038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e70c9f15f44c519b6e7def43b2dd3d" candidate_uuid: "21eac8d83b5b4a2bb11061763bd82b6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afa5537dc2ff4fce979f4664bc607ee5"
I20260926 16:33:02.743794 17761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e70c9f15f44c519b6e7def43b2dd3d" candidate_uuid: "21eac8d83b5b4a2bb11061763bd82b6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb633f392ea64bdc8efc2d5e442c6f44"
I20260926 16:33:02.743920 18038 raft_consensus.cc:3037] T 39e70c9f15f44c519b6e7def43b2dd3d P afa5537dc2ff4fce979f4664bc607ee5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:33:02.744100 17761 raft_consensus.cc:3037] T 39e70c9f15f44c519b6e7def43b2dd3d P cb633f392ea64bdc8efc2d5e442c6f44 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:33:02.746410 17761 raft_consensus.cc:2454] T 39e70c9f15f44c519b6e7def43b2dd3d P cb633f392ea64bdc8efc2d5e442c6f44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21eac8d83b5b4a2bb11061763bd82b6f in term 1.
I20260926 16:33:02.746477 18038 raft_consensus.cc:2454] T 39e70c9f15f44c519b6e7def43b2dd3d P afa5537dc2ff4fce979f4664bc607ee5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21eac8d83b5b4a2bb11061763bd82b6f in term 1.
I20260926 16:33:02.746896 17840 leader_election.cc:302] T 39e70c9f15f44c519b6e7def43b2dd3d P 21eac8d83b5b4a2bb11061763bd82b6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21eac8d83b5b4a2bb11061763bd82b6f, cb633f392ea64bdc8efc2d5e442c6f44; no voters:
I20260926 16:33:02.747221 18108 raft_consensus.cc:2781] T 39e70c9f15f44c519b6e7def43b2dd3d P 21eac8d83b5b4a2bb11061763bd82b6f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:33:02.747741 18108 raft_consensus.cc:686] T 39e70c9f15f44c519b6e7def43b2dd3d P 21eac8d83b5b4a2bb11061763bd82b6f [term 1 LEADER]: Becoming Leader. State: Replica: 21eac8d83b5b4a2bb11061763bd82b6f, State: Running, Role: LEADER
I20260926 16:33:02.748281 18108 consensus_queue.cc:237] T 39e70c9f15f44c519b6e7def43b2dd3d P 21eac8d83b5b4a2bb11061763bd82b6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eac8d83b5b4a2bb11061763bd82b6f" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 38071 } } peers { permanent_uuid: "afa5537dc2ff4fce979f4664bc607ee5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 36785 } } peers { permanent_uuid: "cb633f392ea64bdc8efc2d5e442c6f44" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 34579 } }
I20260926 16:33:02.753288 17593 catalog_manager.cc:5495] T 39e70c9f15f44c519b6e7def43b2dd3d P 21eac8d83b5b4a2bb11061763bd82b6f reported cstate change: term changed from 0 to 1, leader changed from <none> to 21eac8d83b5b4a2bb11061763bd82b6f (127.6.33.2). New cstate: current_term: 1 leader_uuid: "21eac8d83b5b4a2bb11061763bd82b6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eac8d83b5b4a2bb11061763bd82b6f" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 38071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "afa5537dc2ff4fce979f4664bc607ee5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 36785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb633f392ea64bdc8efc2d5e442c6f44" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 34579 } health_report { overall_health: UNKNOWN } } }
I20260926 16:33:02.813187 18097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:33:02.807338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33792 (local address 127.6.33.2:38071)
0926 16:33:02.807540 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 16:33:02.807544 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:33:02.807601 (+ 57us) server_negotiation.cc:407] Connection header received
0926 16:33:02.807784 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:33:02.807788 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:33:02.807878 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:33:02.808015 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:33:02.808378 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:02.809255 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:02.810811 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:02.811217 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:02.811321 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:33:02.811827 (+ 506us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:33:02.812447 (+ 620us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:33:02.812588 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 16:33:02.812826 (+ 238us) server_negotiation.cc:299] Negotiation successful
0926 16:33:02.812912 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 16:33:02.821938 17761 raft_consensus.cc:1261] T 39e70c9f15f44c519b6e7def43b2dd3d P cb633f392ea64bdc8efc2d5e442c6f44 [term 1 FOLLOWER]: Refusing update from remote peer 21eac8d83b5b4a2bb11061763bd82b6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:33:02.822069 18038 raft_consensus.cc:1261] T 39e70c9f15f44c519b6e7def43b2dd3d P afa5537dc2ff4fce979f4664bc607ee5 [term 1 FOLLOWER]: Refusing update from remote peer 21eac8d83b5b4a2bb11061763bd82b6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:33:02.823176 18113 consensus_queue.cc:1035] T 39e70c9f15f44c519b6e7def43b2dd3d P 21eac8d83b5b4a2bb11061763bd82b6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb633f392ea64bdc8efc2d5e442c6f44" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 34579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:33:02.823711 18108 consensus_queue.cc:1035] T 39e70c9f15f44c519b6e7def43b2dd3d P 21eac8d83b5b4a2bb11061763bd82b6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "afa5537dc2ff4fce979f4664bc607ee5" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 36785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:33:02.862124 17592 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:33:02.862445 17592 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:33:02.866508 17592 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f34b2bc957b8474fa4bcb5ff02ffd06b: Sending DeleteTablet for 3 replicas of tablet 39e70c9f15f44c519b6e7def43b2dd3d
I20260926 16:33:02.867847 17884 tablet_service.cc:1514] Processing DeleteTablet for tablet 39e70c9f15f44c519b6e7def43b2dd3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:33:02 UTC) from {username='kudu', principal='kudu/127.6.33.62@KRBTEST.COM'} at 127.0.0.1:33784
I20260926 16:33:02.867995 17741 tablet_service.cc:1514] Processing DeleteTablet for tablet 39e70c9f15f44c519b6e7def43b2dd3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:33:02 UTC) from {username='kudu', principal='kudu/127.6.33.62@KRBTEST.COM'} at 127.0.0.1:46288
I20260926 16:33:02.868690 18018 tablet_service.cc:1514] Processing DeleteTablet for tablet 39e70c9f15f44c519b6e7def43b2dd3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:33:02 UTC) from {username='kudu', principal='kudu/127.6.33.62@KRBTEST.COM'} at 127.0.0.1:59916
I20260926 16:33:02.869009 18126 tablet_replica.cc:330] T 39e70c9f15f44c519b6e7def43b2dd3d P cb633f392ea64bdc8efc2d5e442c6f44: stopping tablet replica
I20260926 16:33:02.869023 18125 tablet_replica.cc:330] T 39e70c9f15f44c519b6e7def43b2dd3d P 21eac8d83b5b4a2bb11061763bd82b6f: stopping tablet replica
I20260926 16:33:02.869444 18126 raft_consensus.cc:2229] T 39e70c9f15f44c519b6e7def43b2dd3d P cb633f392ea64bdc8efc2d5e442c6f44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:33:02.869652 18125 raft_consensus.cc:2229] T 39e70c9f15f44c519b6e7def43b2dd3d P 21eac8d83b5b4a2bb11061763bd82b6f [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:33:02.870008 18126 raft_consensus.cc:2258] T 39e70c9f15f44c519b6e7def43b2dd3d P cb633f392ea64bdc8efc2d5e442c6f44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:33:02.870569 18125 raft_consensus.cc:2258] T 39e70c9f15f44c519b6e7def43b2dd3d P 21eac8d83b5b4a2bb11061763bd82b6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:33:02.871143 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 17673
I20260926 16:33:02.873011 18126 ts_tablet_manager.cc:1905] T 39e70c9f15f44c519b6e7def43b2dd3d P cb633f392ea64bdc8efc2d5e442c6f44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:33:02.873951 18125 ts_tablet_manager.cc:1905] T 39e70c9f15f44c519b6e7def43b2dd3d P 21eac8d83b5b4a2bb11061763bd82b6f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:33:02.880009 18127 tablet_replica.cc:330] T 39e70c9f15f44c519b6e7def43b2dd3d P afa5537dc2ff4fce979f4664bc607ee5: stopping tablet replica
I20260926 16:33:02.880493 18127 raft_consensus.cc:2229] T 39e70c9f15f44c519b6e7def43b2dd3d P afa5537dc2ff4fce979f4664bc607ee5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:33:02.881076 18127 raft_consensus.cc:2258] T 39e70c9f15f44c519b6e7def43b2dd3d P afa5537dc2ff4fce979f4664bc607ee5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:33:02.882040 18126 ts_tablet_manager.cc:1918] T 39e70c9f15f44c519b6e7def43b2dd3d P cb633f392ea64bdc8efc2d5e442c6f44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:33:02.882222 18126 log.cc:1196] T 39e70c9f15f44c519b6e7def43b2dd3d P cb633f392ea64bdc8efc2d5e442c6f44: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ts-0/wal/wals/39e70c9f15f44c519b6e7def43b2dd3d
I20260926 16:33:02.882753 18125 ts_tablet_manager.cc:1918] T 39e70c9f15f44c519b6e7def43b2dd3d P 21eac8d83b5b4a2bb11061763bd82b6f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:33:02.882867 18126 ts_tablet_manager.cc:1939] T 39e70c9f15f44c519b6e7def43b2dd3d P cb633f392ea64bdc8efc2d5e442c6f44: Deleting consensus metadata
I20260926 16:33:02.882923 18125 log.cc:1196] T 39e70c9f15f44c519b6e7def43b2dd3d P 21eac8d83b5b4a2bb11061763bd82b6f: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ts-1/wal/wals/39e70c9f15f44c519b6e7def43b2dd3d
I20260926 16:33:02.883452 18125 ts_tablet_manager.cc:1939] T 39e70c9f15f44c519b6e7def43b2dd3d P 21eac8d83b5b4a2bb11061763bd82b6f: Deleting consensus metadata
I20260926 16:33:02.884105 18127 ts_tablet_manager.cc:1905] T 39e70c9f15f44c519b6e7def43b2dd3d P afa5537dc2ff4fce979f4664bc607ee5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:33:02.884931 17580 catalog_manager.cc:4841] TS cb633f392ea64bdc8efc2d5e442c6f44 (127.6.33.1:34579): tablet 39e70c9f15f44c519b6e7def43b2dd3d (table test-table [id=e2e86caa01ea4699901db0ef75b3861b]) successfully deleted
I20260926 16:33:02.885294 17580 catalog_manager.cc:4841] TS 21eac8d83b5b4a2bb11061763bd82b6f (127.6.33.2:38071): tablet 39e70c9f15f44c519b6e7def43b2dd3d (table test-table [id=e2e86caa01ea4699901db0ef75b3861b]) successfully deleted
I20260926 16:33:02.890627 18127 ts_tablet_manager.cc:1918] T 39e70c9f15f44c519b6e7def43b2dd3d P afa5537dc2ff4fce979f4664bc607ee5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:33:02.890816 18127 log.cc:1196] T 39e70c9f15f44c519b6e7def43b2dd3d P afa5537dc2ff4fce979f4664bc607ee5: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440233850748-6276-0/minicluster-data/ts-2/wal/wals/39e70c9f15f44c519b6e7def43b2dd3d
I20260926 16:33:02.891222 18127 ts_tablet_manager.cc:1939] T 39e70c9f15f44c519b6e7def43b2dd3d P afa5537dc2ff4fce979f4664bc607ee5: Deleting consensus metadata
I20260926 16:33:02.892741 17580 catalog_manager.cc:4841] TS afa5537dc2ff4fce979f4664bc607ee5 (127.6.33.3:36785): tablet 39e70c9f15f44c519b6e7def43b2dd3d (table test-table [id=e2e86caa01ea4699901db0ef75b3861b]) successfully deleted
Sep 26 16:33:02 dist-test-slave-khd3 krb5kdc[17117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.1@KRBTEST.COM
I20260926 16:33:02.900949 18101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:33:02.873661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46302 (local address 127.6.33.1:34579)
0926 16:33:02.873831 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 16:33:02.873838 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:33:02.881966 (+ 8128us) server_negotiation.cc:407] Connection header received
0926 16:33:02.882185 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:33:02.882190 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:33:02.882326 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:33:02.882453 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:33:02.885692 (+ 3239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:02.886797 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:02.888928 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:02.889585 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:02.889721 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:33:02.894712 (+ 4991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:33:02.894727 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:33:02.894742 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:33:02.894778 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:33:02.897983 (+ 3205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:33:02.898729 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:33:02.898741 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:33:02.898749 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:33:02.898878 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:33:02.899273 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:33:02.899278 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:33:02.899283 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:33:02.899650 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:33:02.899964 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0926 16:33:02.900264 (+ 300us) server_negotiation.cc:299] Negotiation successful
0926 16:33:02.900495 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 16:33:02.901650 17801 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:33:02.993212 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 17673
I20260926 16:33:03.012086 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 17812
Sep 26 16:33:03 dist-test-slave-khd3 krb5kdc[17117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.2@KRBTEST.COM
I20260926 16:33:03.029511 18097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:33:03.012953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33802 (local address 127.6.33.2:38071)
0926 16:33:03.013188 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0926 16:33:03.013193 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:33:03.013214 (+ 21us) server_negotiation.cc:407] Connection header received
0926 16:33:03.013408 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:33:03.013413 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:33:03.013486 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:33:03.013610 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:33:03.014918 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:03.015798 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:03.017341 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:03.017782 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:03.017968 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:33:03.022971 (+ 5003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:33:03.022988 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:33:03.023004 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:33:03.023051 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:33:03.026382 (+ 3331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:33:03.027158 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:33:03.027166 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:33:03.027169 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:33:03.027327 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:33:03.027740 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:33:03.027744 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:33:03.027746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:33:03.028113 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:33:03.028463 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0926 16:33:03.028925 (+ 462us) server_negotiation.cc:299] Negotiation successful
0926 16:33:03.029161 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260926 16:33:03.030165 17944 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:33:03.145411 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 17812
I20260926 16:33:03.163733 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 17955
Sep 26 16:33:03 dist-test-slave-khd3 krb5kdc[17117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.3@KRBTEST.COM
I20260926 16:33:03.179867 18099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:33:03.164565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59928 (local address 127.6.33.3:36785)
0926 16:33:03.164794 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 16:33:03.164801 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:33:03.164822 (+ 21us) server_negotiation.cc:407] Connection header received
0926 16:33:03.165017 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:33:03.165022 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:33:03.165094 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:33:03.165200 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:33:03.166584 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:03.167310 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:03.168669 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:03.169210 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:03.169348 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:33:03.174056 (+ 4708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:33:03.174071 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:33:03.174087 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:33:03.174122 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:33:03.177100 (+ 2978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:33:03.177819 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:33:03.177826 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:33:03.177828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:33:03.177964 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:33:03.178494 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:33:03.178498 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:33:03.178501 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:33:03.178889 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:33:03.179184 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0926 16:33:03.179438 (+ 254us) server_negotiation.cc:299] Negotiation successful
0926 16:33:03.179612 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 16:33:03.180656 18078 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:33:03.271780 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 17955
I20260926 16:33:03.288211 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 17561
I20260926 16:33:03.289573 17623 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:33:03.402766 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 17561
I20260926 16:33:03.423893 6276 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 16:33:03.763268 6276 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 16:33:03.763500 6276 mini_ranger.cc:66] Stopping Ranger...
I20260926 16:33:04.109483 6276 mini_ranger.cc:68] Stopped Ranger
2026-09-26 16:33:04.109 UTC [17147] LOG: received smart shutdown request
2026-09-26 16:33:04.113 UTC [17147] LOG: background worker "logical replication launcher" (PID 17156) exited with exit code 1
2026-09-26 16:33:04.113 UTC [17151] LOG: shutting down
2026-09-26 16:33:04.305 UTC [17147] LOG: database system is shut down
2026-09-26T16:33:04Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (51733 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:33:04 dist-test-slave-khd3 krb5kdc[18141](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:33:04 dist-test-slave-khd3 krb5kdc[18141](info): set up 2 sockets
Sep 26 16:33:04 dist-test-slave-khd3 krb5kdc[18141](info): commencing operation
krb5kdc: starting...
W20260926 16:33:05.562086 6276 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Sep 26 16:33:05 dist-test-slave-khd3 krb5kdc[18141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:33:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:33:05Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.6.33.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.6.33.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.6.33.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/krb5kdc/rangeradmin_127.6.33.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.6.33.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/krb5kdc/rangeradmin_127.6.33.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.6.33.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.6.33.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.6.33.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/krb5kdc/rangerlookup_127.6.33.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.6.33.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/krb5kdc/rangerlookup_127.6.33.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.33.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.33.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.33.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/krb5kdc/HTTP_127.6.33.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.33.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/krb5kdc/HTTP_127.6.33.20@KRBTEST.COM.keytab.
I20260926 16:33:05.893469 6276 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskVxavWi/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/postgres -l logfile start
2026-09-26 16:33:08.483 UTC [18171] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 16:33:08.483 UTC [18171] LOG: listening on IPv4 address "127.6.33.20", port 57079
2026-09-26 16:33:08.486 UTC [18171] LOG: listening on Unix socket "/tmp/.s.PGSQL.57079"
2026-09-26 16:33:08.509 UTC [18174] LOG: database system was shut down at 2026-09-26 16:33:06 UTC
2026-09-26 16:33:08.512 UTC [18171] LOG: database system is ready to accept connections
I20260926 16:33:09.485685 6276 mini_postgres.cc:96] Postgres bound to 57079
2026-09-26 16:33:09.502 UTC [18182] FATAL: database "slave" does not exist
127.6.33.20:57079 - accepting connections
I20260926 16:33:09.503086 6276 mini_ranger.cc:161] Starting Ranger...
I20260926 16:33:09.529489 6276 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 16:33:10.042220 6276 mini_ranger.cc:87] Created ranger Postgres database
I20260926 16:33:10.042316 6276 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ranger-admin
2026-09-26 16:33:10,337 [I] DB FLAVOR :POSTGRES
2026-09-26 16:33:10,337 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 16:33:10,337 [I] Checking connection..
2026-09-26 16:33:10,337 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxavWi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVxavWi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.33.20:57079/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 16:33:10,590 [I] Checking connection passed.
2026-09-26 16:33:10,590 [I] --------- Verifying version history table ---------
2026-09-26 16:33:10,590 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxavWi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVxavWi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.33.20:57079/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 16:33:10,856 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 16:33:10,857 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxavWi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVxavWi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.33.20:57079/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 16:33:11,124 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 16:33:11,124 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 16:33:11,124 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxavWi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVxavWi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.33.20:57079/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskVxavWi/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 16:33:11.365 UTC [18246] WARNING: there is no transaction in progress
2026-09-26 16:33:11,372 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 16:33:11,372 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxavWi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVxavWi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.33.20:57079/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 16:33:11,643 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 16:33:11,643 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 16:33:11,644 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxavWi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVxavWi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.33.20:57079/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 16:33:11,905 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxavWi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVxavWi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.33.20:57079/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 16:33:12,151 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxavWi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVxavWi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.33.20:57079/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-khd3.c.gcp-upstream.internal','N') ;"
2026-09-26 16:33:12,394 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxavWi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVxavWi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.33.20:57079/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 16:33:12,657 [I] Table x_portal_user does not exist in database ranger
2026-09-26 16:33:12,658 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxavWi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVxavWi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.33.20:57079/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 16:33:12,949 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 16:33:12,950 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxavWi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVxavWi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.33.20:57079/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 16:33:13,203 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxavWi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVxavWi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.33.20:57079/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 16:33:13,449 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 16:33:13,450 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxavWi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVxavWi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.33.20:57079/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskVxavWi/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 16:33:14.464 UTC [18354] WARNING: there is no transaction in progress
2026-09-26 16:33:14.480 UTC [18354] WARNING: there is no transaction in progress
2026-09-26 16:33:14.497 UTC [18354] WARNING: there is no transaction in progress
2026-09-26 16:33:14.512 UTC [18354] WARNING: there is no transaction in progress
2026-09-26 16:33:14.528 UTC [18354] WARNING: there is no transaction in progress
2026-09-26 16:33:14.547 UTC [18354] WARNING: there is no transaction in progress
2026-09-26 16:33:14.605 UTC [18354] WARNING: there is no transaction in progress
2026-09-26 16:33:14.613 UTC [18354] WARNING: there is no transaction in progress
2026-09-26 16:33:14.629 UTC [18354] WARNING: there is no transaction in progress
2026-09-26 16:33:14.640 UTC [18354] WARNING: there is no transaction in progress
2026-09-26 16:33:14.651 UTC [18354] WARNING: there is no transaction in progress
2026-09-26 16:33:14.666 UTC [18354] WARNING: there is no transaction in progress
2026-09-26 16:33:14.676 UTC [18354] WARNING: there is no transaction in progress
2026-09-26 16:33:14.683 UTC [18354] WARNING: there is no transaction in progress
2026-09-26 16:33:14.691 UTC [18354] WARNING: there is no transaction in progress
2026-09-26 16:33:14.698 UTC [18354] WARNING: there is no transaction in progress
2026-09-26 16:33:15,434 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 16:33:15,434 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxavWi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVxavWi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.33.20:57079/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 16:33:15,692 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 16:33:15,692 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxavWi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVxavWi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.33.20:57079/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 16:33:15,937 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 16:33:15,938 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxavWi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVxavWi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.33.20:57079/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 16:33:16,199 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 16:33:16,199 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxavWi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVxavWi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.33.20:57079/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 16:33:16,459 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxavWi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVxavWi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.33.20:57079/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 16:33:16,722 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxavWi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVxavWi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.33.20:57079/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-khd3.c.gcp-upstream.internal';"
2026-09-26 16:33:16,979 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 16:33:16,980 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxavWi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVxavWi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.33.20:57079/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 16:33:17,220 [I] DB_PATCHES have already been applied
I20260926 16:33:17.226420 6276 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ranger-admin:/tmp/dist-test-taskVxavWi/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskVxavWi/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskVxavWi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVxavWi/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 16:33:17.226507 6276 mini_ranger.cc:193] Using host: 127.6.33.20
I20260926 16:33:17.241624 6276 mini_ranger.cc:227] Ranger admin URL: http://127.6.33.20:35815
Sep 26, 2026 4:33:17 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 4:33:17 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 4:33:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 4:33:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 4:33:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 4:33:18 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 4:33:18 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 16:33:18.372313 6276 mini_ranger.cc:160] Time spent starting Ranger: real 8.869s user 0.000s sys 0.039s
Sep 26, 2026 4:33:38 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 4:33:38 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 4:33:38 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 4:33:38 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 4:33:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 4:33:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 4:33:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 4:33:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 4:33:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 4:33:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 4:33:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 4:33:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 4:33:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 4:33:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 4:33:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 4:33:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 4:33:38 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 4:33:39 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 16:33:39.990833 6276 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.6.33.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.6.33.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.6.33.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/krb5kdc/rangerkms_127.6.33.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.6.33.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/krb5kdc/rangerkms_127.6.33.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.33.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.6.33.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.33.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/krb5kdc/HTTP_127.6.33.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.33.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/krb5kdc/HTTP_127.6.33.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 16:33:40 dist-test-slave-khd3 krb5kdc[18141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440420, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 16:33:40.158749 6276 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 16:33:40.179970 6276 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 16:33:41.102444 6276 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 16:33:41.102574 6276 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ranger-kms
I20260926 16:33:41.102622 6276 mini_ranger_kms.cc:227] Using postgres at 127.6.33.20:57079
2026-09-26 16:33:41,465 [I] DB FLAVOR :POSTGRES
2026-09-26 16:33:41,466 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 16:33:41,466 [I] Checking connection
2026-09-26 16:33:41,466 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxavWi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.33.20:57079/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 16:33:41,725 [I] connection success
2026-09-26 16:33:41,725 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 16:33:41,725 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 16:33:41,725 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxavWi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.33.20:57079/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 16:33:41,961 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 16:33:41,961 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 16:33:41,961 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 16:33:41,961 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxavWi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.33.20:57079/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 16:33:42,234 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 16:33:42.738744 6276 mini_ranger_kms.cc:310] Created kms service
I20260926 16:33:42.954389 6276 mini_ranger_kms.cc:326] Created kudu user
I20260926 16:33:42.998935 6276 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 16:33:42 dist-test-slave-khd3 krb5kdc[18141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440420, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.6.33.20@KRBTEST.COM
I20260926 16:33:43.397395 6276 mini_ranger_kms.cc:384] Added ranger policy
I20260926 16:33:43.397737 6276 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ranger-kms:/tmp/dist-test-taskVxavWi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVxavWi/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskVxavWi/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 16:33:43.397791 6276 mini_ranger_kms.cc:240] Using host: 127.6.33.20
I20260926 16:33:43.414587 6276 mini_ranger_kms.cc:276] Ranger KMS PID: 18534
I20260926 16:33:43.414717 6276 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.6.33.20:46791
Sep 26, 2026 4:33:44 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 4:33:44 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 4:33:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 4:33:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 4:33:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 4:33:45 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 16:33:45.786094 6276 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.627s user 0.099s sys 0.281s
I20260926 16:33:45.786350 6276 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 16:33:45.786530 6276 mini_ranger_kms.cc:401] 127.6.33.20:46791/kms/v1/keys
Sep 26 16:33:45 dist-test-slave-khd3 krb5kdc[18141](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790440425, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.33.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 16:33:46 dist-test-slave-khd3 krb5kdc[18141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440425, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.33.20@KRBTEST.COM for HTTP/127.6.33.20@KRBTEST.COM
Sep 26 16:33:46 dist-test-slave-khd3 krb5kdc[18141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440425, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.33.20@KRBTEST.COM for HTTP/127.6.33.20@KRBTEST.COM
Sep 26 16:33:48 dist-test-slave-khd3 krb5kdc[18141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.6.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab.
Entry for principal kudu/127.6.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab.
I20260926 16:33:49.033520 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:46825
--webserver_interface=127.6.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:36459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.33.62:46825
--ranger_config_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.33.20:46791/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:33:49.372056 18585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:33:49.372355 18585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:33:49.372536 18585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:33:49.380169 18585 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:33:49.380297 18585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:33:49.380388 18585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:33:49.380472 18585 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:33:49.380564 18585 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:33:49.391049 18585 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:36459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/master-0/wal
--ranger_kms_url=127.6.33.20:46791/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.6.33.62:46825
--ranger_config_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:46825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:33:49.392447 18585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:33:49.393918 18585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:33:49.412164 18593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:33:49.412578 18585 server_base.cc:1034] running on GCE node
W20260926 16:33:49.412176 18590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:33:49.412154 18591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:33:49.413846 18585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:33:49.415390 18585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:33:49.416616 18585 hybrid_clock.cc:648] HybridClock initialized: now 1790440429416560 us; error 46 us; skew 500 ppm
I20260926 16:33:49.416998 18585 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:33:49 dist-test-slave-khd3 krb5kdc[18141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440429, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:33:49.432399 18585 init.cc:377] Logged in from keytab as kudu/127.6.33.62@KRBTEST.COM (short username kudu)
I20260926 16:33:49.434250 18585 webserver.cc:460] Webserver started at http://127.6.33.62:43693/ using document root <none> and password file <none>
I20260926 16:33:49.434983 18585 fs_manager.cc:359] Metadata directory not provided
I20260926 16:33:49.435117 18585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:33:49.435458 18585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 16:33:49 dist-test-slave-khd3 krb5kdc[18141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440429, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.62@KRBTEST.COM for HTTP/127.6.33.20@KRBTEST.COM
I20260926 16:33:49.486009 18585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/master-0/data/instance:
uuid: "8e5a8d2f6bda4ca48afeac012546a6aa"
format_stamp: "Formatted at 2026-09-26 16:33:49 on dist-test-slave-khd3"
server_key: "d380bcd9d977c4c136c776ccf7244cd5"
server_key_iv: "d17ad1749380495726aa1b87976f44c9"
server_key_version: "kuduclusterkey@0"
I20260926 16:33:49.486913 18585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/master-0/wal/instance:
uuid: "8e5a8d2f6bda4ca48afeac012546a6aa"
format_stamp: "Formatted at 2026-09-26 16:33:49 on dist-test-slave-khd3"
server_key: "d380bcd9d977c4c136c776ccf7244cd5"
server_key_iv: "d17ad1749380495726aa1b87976f44c9"
server_key_version: "kuduclusterkey@0"
I20260926 16:33:49.492528 18585 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 16:33:49.509831 18600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:33:49.511523 18585 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260926 16:33:49.511726 18585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/master-0/wal
uuid: "8e5a8d2f6bda4ca48afeac012546a6aa"
format_stamp: "Formatted at 2026-09-26 16:33:49 on dist-test-slave-khd3"
server_key: "d380bcd9d977c4c136c776ccf7244cd5"
server_key_iv: "d17ad1749380495726aa1b87976f44c9"
server_key_version: "kuduclusterkey@0"
I20260926 16:33:49.511981 18585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:33:49.530481 18585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:33:49.534783 18585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:33:49.535135 18585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:33:49.554239 18585 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.62:46825
I20260926 16:33:49.554276 18651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.62:46825 every 8 connection(s)
I20260926 16:33:49.555644 18585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
I20260926 16:33:49.558202 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 18585
I20260926 16:33:49.558430 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/master-0/wal/instance
I20260926 16:33:49.559350 6276 external_mini_cluster.cc:1428] Setting key f9aa96f3f35deeeb1ced5ce6dd0e66ff
I20260926 16:33:49.559860 18652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:33:49.573331 18652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e5a8d2f6bda4ca48afeac012546a6aa: Bootstrap starting.
Sep 26 16:33:49 dist-test-slave-khd3 krb5kdc[18141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:33:49.577375 18652 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e5a8d2f6bda4ca48afeac012546a6aa: Neither blocks nor log segments found. Creating new log.
I20260926 16:33:49.578840 18652 log.cc:824] T 00000000000000000000000000000000 P 8e5a8d2f6bda4ca48afeac012546a6aa: Log is configured to *not* fsync() on all Append() calls
I20260926 16:33:49.583178 18652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e5a8d2f6bda4ca48afeac012546a6aa: No bootstrap required, opened a new log
I20260926 16:33:49.585353 18655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:33:49.561631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45820 (local address 127.6.33.62:46825)
0926 16:33:49.563168 (+ 1537us) server_negotiation.cc:206] Beginning negotiation
0926 16:33:49.563187 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 16:33:49.563556 (+ 369us) server_negotiation.cc:407] Connection header received
0926 16:33:49.565407 (+ 1851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:33:49.565430 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:33:49.565856 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:33:49.566303 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:33:49.567675 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:49.569039 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:49.570379 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:49.571134 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:49.571357 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:33:49.575804 (+ 4447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:33:49.575817 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:33:49.575839 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:33:49.575895 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:33:49.580002 (+ 4107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:33:49.580945 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:33:49.580956 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:33:49.580967 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:33:49.581194 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:33:49.581700 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:33:49.581709 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:33:49.581713 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:33:49.582521 (+ 808us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:33:49.582837 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0926 16:33:49.583318 (+ 481us) server_negotiation.cc:299] Negotiation successful
0926 16:33:49.583813 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1242,"thread_start_us":1008,"threads_started":1}
I20260926 16:33:49.593174 18652 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e5a8d2f6bda4ca48afeac012546a6aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e5a8d2f6bda4ca48afeac012546a6aa" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 46825 } }
I20260926 16:33:49.593552 18652 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e5a8d2f6bda4ca48afeac012546a6aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:33:49.593640 18652 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e5a8d2f6bda4ca48afeac012546a6aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e5a8d2f6bda4ca48afeac012546a6aa, State: Initialized, Role: FOLLOWER
I20260926 16:33:49.594249 18652 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e5a8d2f6bda4ca48afeac012546a6aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e5a8d2f6bda4ca48afeac012546a6aa" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 46825 } }
I20260926 16:33:49.594470 18652 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e5a8d2f6bda4ca48afeac012546a6aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:33:49.594580 18652 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e5a8d2f6bda4ca48afeac012546a6aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:33:49.594717 18652 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e5a8d2f6bda4ca48afeac012546a6aa [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:33:49.597352 18652 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e5a8d2f6bda4ca48afeac012546a6aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e5a8d2f6bda4ca48afeac012546a6aa" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 46825 } }
I20260926 16:33:49.597829 18652 leader_election.cc:302] T 00000000000000000000000000000000 P 8e5a8d2f6bda4ca48afeac012546a6aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e5a8d2f6bda4ca48afeac012546a6aa; no voters:
I20260926 16:33:49.598423 18652 leader_election.cc:288] T 00000000000000000000000000000000 P 8e5a8d2f6bda4ca48afeac012546a6aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:33:49.598605 18657 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e5a8d2f6bda4ca48afeac012546a6aa [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:33:49.599339 18657 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e5a8d2f6bda4ca48afeac012546a6aa [term 1 LEADER]: Becoming Leader. State: Replica: 8e5a8d2f6bda4ca48afeac012546a6aa, State: Running, Role: LEADER
I20260926 16:33:49.599833 18657 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e5a8d2f6bda4ca48afeac012546a6aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e5a8d2f6bda4ca48afeac012546a6aa" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 46825 } }
I20260926 16:33:49.600778 18652 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e5a8d2f6bda4ca48afeac012546a6aa [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:33:49.603150 18659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e5a8d2f6bda4ca48afeac012546a6aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e5a8d2f6bda4ca48afeac012546a6aa. Latest consensus state: current_term: 1 leader_uuid: "8e5a8d2f6bda4ca48afeac012546a6aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e5a8d2f6bda4ca48afeac012546a6aa" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 46825 } } }
I20260926 16:33:49.603493 18659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e5a8d2f6bda4ca48afeac012546a6aa [sys.catalog]: This master's current role is: LEADER
I20260926 16:33:49.604388 18663 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:33:49.604218 18658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e5a8d2f6bda4ca48afeac012546a6aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e5a8d2f6bda4ca48afeac012546a6aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e5a8d2f6bda4ca48afeac012546a6aa" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 46825 } } }
I20260926 16:33:49.608505 18658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e5a8d2f6bda4ca48afeac012546a6aa [sys.catalog]: This master's current role is: LEADER
I20260926 16:33:49.611671 18663 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:33:49.619683 18652 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 16:33:49.635701 18663 catalog_manager.cc:1347] Generated new cluster ID: af54450b8ff749909bea7b8754d36b6f
I20260926 16:33:49.635953 18663 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:33:49.681632 18663 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:33:49.684082 18663 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:33:49.697734 18663 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e5a8d2f6bda4ca48afeac012546a6aa: Generated new TSK 0
I20260926 16:33:49.698921 18663 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.33.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab.
Entry for principal kudu/127.6.33.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab.
I20260926 16:33:50.416534 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.33.20:46791/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.1:0
--local_ip_for_outbound_sockets=127.6.33.1
--webserver_interface=127.6.33.1
--webserver_port=0
--tserver_master_addrs=127.6.33.62:46825
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab
--principal=kudu/127.6.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:36459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 16:33:50 dist-test-slave-khd3 krb5kdc[18141](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790440430, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 16:33:50.909312 18697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:33:50.909780 18697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:33:50.909951 18697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:33:50.921854 18697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:33:50.921991 18697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:33:50.922185 18697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.1
I20260926 16:33:50.932451 18697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:36459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.6.33.20:46791/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab
--principal=kudu/127.6.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.33.1
--webserver_port=0
--tserver_master_addrs=127.6.33.62:46825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.1
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:33:50.933897 18697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:33:50.935802 18697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:33:50.962056 18708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:33:50.967283 18707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:33:50.967567 18697 server_base.cc:1034] running on GCE node
W20260926 16:33:50.968132 18710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:33:50.969033 18697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:33:50.970422 18697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:33:50.971693 18697 hybrid_clock.cc:648] HybridClock initialized: now 1790440430971624 us; error 59 us; skew 500 ppm
I20260926 16:33:50.972128 18697 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:33:50 dist-test-slave-khd3 krb5kdc[18141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440430, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:33:50.976595 18697 init.cc:377] Logged in from keytab as kudu/127.6.33.1@KRBTEST.COM (short username kudu)
I20260926 16:33:50.987133 18697 webserver.cc:460] Webserver started at http://127.6.33.1:35491/ using document root <none> and password file <none>
I20260926 16:33:50.988145 18697 fs_manager.cc:359] Metadata directory not provided
I20260926 16:33:50.988294 18697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:33:50.988639 18697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 16:33:50 dist-test-slave-khd3 krb5kdc[18141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440430, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.1@KRBTEST.COM for HTTP/127.6.33.20@KRBTEST.COM
I20260926 16:33:51.013549 18697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-0/data/instance:
uuid: "79be1661dfbb4578b30084cc701559de"
format_stamp: "Formatted at 2026-09-26 16:33:51 on dist-test-slave-khd3"
server_key: "6736a222fa8fa42f365494c416527ef7"
server_key_iv: "9230c8e7e20ec2eee7f432815da84129"
server_key_version: "kuduclusterkey@0"
I20260926 16:33:51.015161 18697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-0/wal/instance:
uuid: "79be1661dfbb4578b30084cc701559de"
format_stamp: "Formatted at 2026-09-26 16:33:51 on dist-test-slave-khd3"
server_key: "6736a222fa8fa42f365494c416527ef7"
server_key_iv: "9230c8e7e20ec2eee7f432815da84129"
server_key_version: "kuduclusterkey@0"
I20260926 16:33:51.038791 18697 fs_manager.cc:691] Time spent creating directory manager: real 0.023s user 0.014s sys 0.008s
I20260926 16:33:51.076867 18717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:33:51.080134 18697 fs_manager.cc:725] Time spent opening block manager: real 0.023s user 0.011s sys 0.009s
I20260926 16:33:51.080364 18697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-0/wal
uuid: "79be1661dfbb4578b30084cc701559de"
format_stamp: "Formatted at 2026-09-26 16:33:51 on dist-test-slave-khd3"
server_key: "6736a222fa8fa42f365494c416527ef7"
server_key_iv: "9230c8e7e20ec2eee7f432815da84129"
server_key_version: "kuduclusterkey@0"
I20260926 16:33:51.080658 18697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:33:51.108851 18697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:33:51.114485 18697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:33:51.114815 18697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:33:51.123399 18697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:33:51.125329 18697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:33:51.125428 18697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:33:51.125559 18697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:33:51.125617 18697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 16:33:51 dist-test-slave-khd3 krb5kdc[18141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440430, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.62@KRBTEST.COM for HTTP/127.6.33.20@KRBTEST.COM
I20260926 16:33:51.237746 18697 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.1:41029
I20260926 16:33:51.237789 18829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.1:41029 every 8 connection(s)
I20260926 16:33:51.239398 18697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
I20260926 16:33:51.241067 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 18697
I20260926 16:33:51.241274 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-0/wal/instance
I20260926 16:33:51.242048 6276 external_mini_cluster.cc:1428] Setting key 4d1c8808d0a58e051c7ebeee3c7854dd
WARNING: no policy specified for kudu/127.6.33.2@KRBTEST.COM; defaulting to no policy
Sep 26 16:33:51 dist-test-slave-khd3 krb5kdc[18141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440430, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.1@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.2@KRBTEST.COM" created.
I20260926 16:33:51.297613 18833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:33:51.255221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:59619 (local address 127.6.33.62:46825)
0926 16:33:51.274676 (+ 19455us) server_negotiation.cc:206] Beginning negotiation
0926 16:33:51.274685 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:33:51.274716 (+ 31us) server_negotiation.cc:407] Connection header received
0926 16:33:51.276128 (+ 1412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:33:51.276138 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:33:51.276270 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:33:51.276400 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:33:51.278941 (+ 2541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:51.280285 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:51.282865 (+ 2580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:51.283688 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:51.283849 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:33:51.289075 (+ 5226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:33:51.289082 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:33:51.289085 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:33:51.289124 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:33:51.292944 (+ 3820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:33:51.294095 (+ 1151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:33:51.294104 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:33:51.294111 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:33:51.294289 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:33:51.294903 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:33:51.294911 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:33:51.294917 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:33:51.295427 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:33:51.295572 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 16:33:51.296820 (+ 1248us) server_negotiation.cc:299] Negotiation successful
0926 16:33:51.297106 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19279,"thread_start_us":19487,"threads_started":1}
I20260926 16:33:51.300168 18834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:33:51.255861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:46825 (local address 127.6.33.1:59619)
0926 16:33:51.270421 (+ 14560us) negotiation.cc:107] Waiting for socket to connect
0926 16:33:51.270483 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 16:33:51.275532 (+ 5049us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:33:51.276971 (+ 1439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:33:51.276987 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:33:51.277709 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:33:51.278705 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:33:51.278737 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:33:51.280506 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:33:51.280514 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:33:51.282685 (+ 2171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:33:51.282699 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:33:51.284228 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:33:51.284239 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:33:51.284605 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:33:51.285529 (+ 924us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:33:51.285576 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:33:51.288749 (+ 3173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:33:51.293176 (+ 4427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:33:51.293190 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:33:51.293215 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:33:51.293922 (+ 707us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:33:51.294458 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:33:51.294465 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:33:51.294472 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:33:51.294765 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:33:51.295682 (+ 917us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:33:51.295694 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:33:51.296301 (+ 607us) client_negotiation.cc:769] Sending connection context
0926 16:33:51.298012 (+ 1711us) client_negotiation.cc:240] Negotiation successful
0926 16:33:51.298664 (+ 652us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14186,"thread_start_us":13964,"threads_started":1}
I20260926 16:33:51.301964 18830 heartbeater.cc:347] Connected to a master server at 127.6.33.62:46825
I20260926 16:33:51.302340 18830 heartbeater.cc:464] Registering TS with master...
I20260926 16:33:51.303256 18830 heartbeater.cc:511] Master 127.6.33.62:46825 requested a full tablet report, sending...
I20260926 16:33:51.305672 18617 ts_manager.cc:194] Registered new tserver with Master: 79be1661dfbb4578b30084cc701559de (127.6.33.1:41029)
I20260926 16:33:51.307245 18617 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.33.1@KRBTEST.COM'} at 127.6.33.1:59619
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab.
Entry for principal kudu/127.6.33.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab.
I20260926 16:33:51.332720 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.33.20:46791/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.2:0
--local_ip_for_outbound_sockets=127.6.33.2
--webserver_interface=127.6.33.2
--webserver_port=0
--tserver_master_addrs=127.6.33.62:46825
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab
--principal=kudu/127.6.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:36459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 16:33:51 dist-test-slave-khd3 krb5kdc[18141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440430, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.62@KRBTEST.COM for HTTP/127.6.33.20@KRBTEST.COM
W20260926 16:33:51.798725 18836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:33:51.799319 18836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:33:51.799526 18836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:33:51.811203 18836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:33:51.811455 18836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:33:51.811717 18836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.2
I20260926 16:33:51.828013 18836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:36459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.6.33.20:46791/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab
--principal=kudu/127.6.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.33.2
--webserver_port=0
--tserver_master_addrs=127.6.33.62:46825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.2
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:33:51.830176 18836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:33:51.832016 18836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:33:51.872598 18844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:33:51.873471 18843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:33:51.882613 18846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:33:51.883253 18836 server_base.cc:1034] running on GCE node
I20260926 16:33:51.884346 18836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:33:51.901469 18836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:33:51.902820 18836 hybrid_clock.cc:648] HybridClock initialized: now 1790440431902709 us; error 123 us; skew 500 ppm
I20260926 16:33:51.903338 18836 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:33:51 dist-test-slave-khd3 krb5kdc[18141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440431, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:33:51.908093 18836 init.cc:377] Logged in from keytab as kudu/127.6.33.2@KRBTEST.COM (short username kudu)
I20260926 16:33:51.921417 18836 webserver.cc:460] Webserver started at http://127.6.33.2:38759/ using document root <none> and password file <none>
I20260926 16:33:51.922610 18836 fs_manager.cc:359] Metadata directory not provided
I20260926 16:33:51.922853 18836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:33:51.923271 18836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 16:33:51 dist-test-slave-khd3 krb5kdc[18141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440431, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.2@KRBTEST.COM for HTTP/127.6.33.20@KRBTEST.COM
I20260926 16:33:51.942914 18836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-1/data/instance:
uuid: "af7963a2edd54c2a9f3e58cad99440cc"
format_stamp: "Formatted at 2026-09-26 16:33:51 on dist-test-slave-khd3"
server_key: "4649cbe4a65c0218e9f7acec40e1dafd"
server_key_iv: "2c0b9b699b880ae908cac63bea960b1c"
server_key_version: "kuduclusterkey@0"
I20260926 16:33:51.944031 18836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-1/wal/instance:
uuid: "af7963a2edd54c2a9f3e58cad99440cc"
format_stamp: "Formatted at 2026-09-26 16:33:51 on dist-test-slave-khd3"
server_key: "4649cbe4a65c0218e9f7acec40e1dafd"
server_key_iv: "2c0b9b699b880ae908cac63bea960b1c"
server_key_version: "kuduclusterkey@0"
I20260926 16:33:51.951608 18836 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20260926 16:33:51.972015 18860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:33:51.974090 18836 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 16:33:51.974323 18836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-1/wal
uuid: "af7963a2edd54c2a9f3e58cad99440cc"
format_stamp: "Formatted at 2026-09-26 16:33:51 on dist-test-slave-khd3"
server_key: "4649cbe4a65c0218e9f7acec40e1dafd"
server_key_iv: "2c0b9b699b880ae908cac63bea960b1c"
server_key_version: "kuduclusterkey@0"
I20260926 16:33:51.974609 18836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:33:52.036288 18836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:33:52.042747 18836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:33:52.043118 18836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:33:52.044402 18836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:33:52.046293 18836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:33:52.046393 18836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:33:52.046571 18836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:33:52.046662 18836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:33:52.102139 18836 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.2:46627
I20260926 16:33:52.102159 18972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.2:46627 every 8 connection(s)
I20260926 16:33:52.103935 18836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
I20260926 16:33:52.112248 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 18836
I20260926 16:33:52.112423 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-1/wal/instance
I20260926 16:33:52.113128 6276 external_mini_cluster.cc:1428] Setting key 6c63e1ce8c762832c3dd86c66acbf0d7
Sep 26 16:33:52 dist-test-slave-khd3 krb5kdc[18141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440431, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.2@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:33:52.135281 18976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:33:52.108599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:48489 (local address 127.6.33.62:46825)
0926 16:33:52.113567 (+ 4968us) server_negotiation.cc:206] Beginning negotiation
0926 16:33:52.113575 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:33:52.113602 (+ 27us) server_negotiation.cc:407] Connection header received
0926 16:33:52.116143 (+ 2541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:33:52.116149 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:33:52.116247 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:33:52.116353 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:33:52.118612 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:52.119714 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:52.122236 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:52.122848 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:52.122951 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:33:52.127563 (+ 4612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:33:52.127570 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:33:52.127575 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:33:52.127617 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:33:52.131309 (+ 3692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:33:52.132331 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:33:52.132338 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:33:52.132342 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:33:52.132507 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:33:52.133092 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:33:52.133098 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:33:52.133102 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:33:52.133456 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:33:52.133592 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 16:33:52.134616 (+ 1024us) server_negotiation.cc:299] Negotiation successful
0926 16:33:52.134866 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4833,"thread_start_us":5749,"threads_started":1}
I20260926 16:33:52.138067 18975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:33:52.107988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:46825 (local address 127.6.33.2:48489)
0926 16:33:52.110421 (+ 2433us) negotiation.cc:107] Waiting for socket to connect
0926 16:33:52.110474 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 16:33:52.115682 (+ 5208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:33:52.116836 (+ 1154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:33:52.116851 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:33:52.117507 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:33:52.118399 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:33:52.118427 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:33:52.119933 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:33:52.119938 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:33:52.122058 (+ 2120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:33:52.122073 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:33:52.123308 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:33:52.123315 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:33:52.123605 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:33:52.124515 (+ 910us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:33:52.124547 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:33:52.127288 (+ 2741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:33:52.131497 (+ 4209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:33:52.131508 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:33:52.131529 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:33:52.132181 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:33:52.132658 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:33:52.132663 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:33:52.132667 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:33:52.132941 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:33:52.133684 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:33:52.133693 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:33:52.134168 (+ 475us) client_negotiation.cc:769] Sending connection context
0926 16:33:52.135883 (+ 1715us) client_negotiation.cc:240] Negotiation successful
0926 16:33:52.136485 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2119,"thread_start_us":1863,"threads_started":1}
I20260926 16:33:52.140228 18973 heartbeater.cc:347] Connected to a master server at 127.6.33.62:46825
I20260926 16:33:52.140561 18973 heartbeater.cc:464] Registering TS with master...
I20260926 16:33:52.141456 18973 heartbeater.cc:511] Master 127.6.33.62:46825 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.33.3@KRBTEST.COM; defaulting to no policy
I20260926 16:33:52.143301 18617 ts_manager.cc:194] Registered new tserver with Master: af7963a2edd54c2a9f3e58cad99440cc (127.6.33.2:46627)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.3@KRBTEST.COM" created.
I20260926 16:33:52.144439 18617 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.33.2@KRBTEST.COM'} at 127.6.33.2:48489
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab.
Entry for principal kudu/127.6.33.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab.
I20260926 16:33:52.178277 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.33.20:46791/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.3:0
--local_ip_for_outbound_sockets=127.6.33.3
--webserver_interface=127.6.33.3
--webserver_port=0
--tserver_master_addrs=127.6.33.62:46825
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab
--principal=kudu/127.6.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:36459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:33:52.311336 18830 heartbeater.cc:503] Master 127.6.33.62:46825 was elected leader, sending a full tablet report...
W20260926 16:33:52.549700 18979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:33:52.550055 18979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:33:52.550199 18979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:33:52.559171 18979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:33:52.559316 18979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:33:52.559487 18979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.3
I20260926 16:33:52.573561 18979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:36459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.6.33.20:46791/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab
--principal=kudu/127.6.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.33.3
--webserver_port=0
--tserver_master_addrs=127.6.33.62:46825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.3
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:33:52.575169 18979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:33:52.576407 18979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:33:52.590430 18984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:33:52.590394 18985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:33:52.590478 18987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:33:52.591568 18979 server_base.cc:1034] running on GCE node
I20260926 16:33:52.592191 18979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:33:52.593353 18979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:33:52.594611 18979 hybrid_clock.cc:648] HybridClock initialized: now 1790440432594560 us; error 48 us; skew 500 ppm
I20260926 16:33:52.595000 18979 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:33:52 dist-test-slave-khd3 krb5kdc[18141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440432, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:33:52.598510 18979 init.cc:377] Logged in from keytab as kudu/127.6.33.3@KRBTEST.COM (short username kudu)
I20260926 16:33:52.600183 18979 webserver.cc:460] Webserver started at http://127.6.33.3:43169/ using document root <none> and password file <none>
I20260926 16:33:52.601039 18979 fs_manager.cc:359] Metadata directory not provided
I20260926 16:33:52.601183 18979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:33:52.601504 18979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 16:33:52 dist-test-slave-khd3 krb5kdc[18141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440432, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.3@KRBTEST.COM for HTTP/127.6.33.20@KRBTEST.COM
I20260926 16:33:52.616331 18979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-2/data/instance:
uuid: "df9078e09f92434eb37ede4920ffb6f9"
format_stamp: "Formatted at 2026-09-26 16:33:52 on dist-test-slave-khd3"
server_key: "0f92b755c3213b8699c9d157f328e8ec"
server_key_iv: "4f831cbadec5294fa1d78def0374cf51"
server_key_version: "kuduclusterkey@0"
I20260926 16:33:52.617343 18979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-2/wal/instance:
uuid: "df9078e09f92434eb37ede4920ffb6f9"
format_stamp: "Formatted at 2026-09-26 16:33:52 on dist-test-slave-khd3"
server_key: "0f92b755c3213b8699c9d157f328e8ec"
server_key_iv: "4f831cbadec5294fa1d78def0374cf51"
server_key_version: "kuduclusterkey@0"
I20260926 16:33:52.623584 18979 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260926 16:33:52.638964 18994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:33:52.640722 18979 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 16:33:52.640990 18979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-2/wal
uuid: "df9078e09f92434eb37ede4920ffb6f9"
format_stamp: "Formatted at 2026-09-26 16:33:52 on dist-test-slave-khd3"
server_key: "0f92b755c3213b8699c9d157f328e8ec"
server_key_iv: "4f831cbadec5294fa1d78def0374cf51"
server_key_version: "kuduclusterkey@0"
I20260926 16:33:52.641234 18979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:33:52.664206 18979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:33:52.668481 18979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:33:52.668838 18979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:33:52.669852 18979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:33:52.671308 18979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:33:52.671419 18979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:33:52.671573 18979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:33:52.671646 18979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:33:52.706310 18979 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.3:45599
I20260926 16:33:52.706326 19106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.3:45599 every 8 connection(s)
I20260926 16:33:52.707634 18979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
I20260926 16:33:52.715751 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 18979
I20260926 16:33:52.715956 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-2/wal/instance
I20260926 16:33:52.716635 6276 external_mini_cluster.cc:1428] Setting key 25b89d7fe90b11acb3e3fb7dd902c2c6
Sep 26 16:33:52 dist-test-slave-khd3 krb5kdc[18141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440432, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.3@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:33:52.729140 19109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:33:52.710452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.3:50209 (local address 127.6.33.62:46825)
0926 16:33:52.710931 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0926 16:33:52.710936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:33:52.711897 (+ 961us) server_negotiation.cc:407] Connection header received
0926 16:33:52.714022 (+ 2125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:33:52.714026 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:33:52.714121 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:33:52.714224 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:33:52.715780 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:52.716996 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:52.718948 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:52.719668 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:52.719796 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:33:52.723291 (+ 3495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:33:52.723296 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:33:52.723299 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:33:52.723332 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:33:52.725999 (+ 2667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:33:52.726752 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:33:52.726764 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:33:52.726766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:33:52.726890 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:33:52.727345 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:33:52.727352 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:33:52.727355 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:33:52.727643 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:33:52.727796 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 16:33:52.728492 (+ 696us) server_negotiation.cc:299] Negotiation successful
0926 16:33:52.728741 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":233,"threads_started":1}
I20260926 16:33:52.730150 19110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:33:52.710977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:46825 (local address 127.6.33.3:50209)
0926 16:33:52.711716 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0926 16:33:52.711760 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 16:33:52.713711 (+ 1951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:33:52.714438 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:33:52.714447 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:33:52.714893 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:33:52.715576 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:33:52.715594 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:33:52.717150 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:33:52.717154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:33:52.718821 (+ 1667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:33:52.718833 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:33:52.719933 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:33:52.719940 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:33:52.720165 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:33:52.720855 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:33:52.720877 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:33:52.723014 (+ 2137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:33:52.726154 (+ 3140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:33:52.726163 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:33:52.726182 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:33:52.726621 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:33:52.727000 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:33:52.727005 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:33:52.727009 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:33:52.727232 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:33:52.727769 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:33:52.727777 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:33:52.728142 (+ 365us) client_negotiation.cc:769] Sending connection context
0926 16:33:52.728424 (+ 282us) client_negotiation.cc:240] Negotiation successful
0926 16:33:52.728992 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":280,"threads_started":1}
I20260926 16:33:52.731743 19107 heartbeater.cc:347] Connected to a master server at 127.6.33.62:46825
I20260926 16:33:52.732155 19107 heartbeater.cc:464] Registering TS with master...
I20260926 16:33:52.733016 19107 heartbeater.cc:511] Master 127.6.33.62:46825 requested a full tablet report, sending...
I20260926 16:33:52.734548 18617 ts_manager.cc:194] Registered new tserver with Master: df9078e09f92434eb37ede4920ffb6f9 (127.6.33.3:45599)
I20260926 16:33:52.735404 18617 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.33.3@KRBTEST.COM'} at 127.6.33.3:50209
I20260926 16:33:52.745906 6276 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:33:52.747730 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 18697
Sep 26 16:33:52 dist-test-slave-khd3 krb5kdc[18141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.1@KRBTEST.COM
I20260926 16:33:52.776898 19112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:33:52.748947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49680 (local address 127.6.33.1:41029)
0926 16:33:52.749595 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0926 16:33:52.749602 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:33:52.749630 (+ 28us) server_negotiation.cc:407] Connection header received
0926 16:33:52.749692 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:33:52.749698 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:33:52.749929 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:33:52.750066 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:33:52.751613 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:52.752746 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:52.754299 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:52.754877 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:52.754979 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:33:52.770297 (+ 15318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:33:52.770317 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:33:52.770334 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:33:52.770384 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:33:52.773673 (+ 3289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:33:52.774546 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:33:52.774555 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:33:52.774557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:33:52.774720 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:33:52.775144 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:33:52.775148 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:33:52.775150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:33:52.775545 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:33:52.775793 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0926 16:33:52.776165 (+ 372us) server_negotiation.cc:299] Negotiation successful
0926 16:33:52.776439 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":315,"threads_started":1}
I20260926 16:33:52.778287 18825 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:33:52.863135 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 18697
I20260926 16:33:52.879609 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 18836
Sep 26 16:33:52 dist-test-slave-khd3 krb5kdc[18141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.2@KRBTEST.COM
I20260926 16:33:52.897378 19114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:33:52.880582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46892 (local address 127.6.33.2:46627)
0926 16:33:52.881160 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0926 16:33:52.881167 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:33:52.881193 (+ 26us) server_negotiation.cc:407] Connection header received
0926 16:33:52.881252 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:33:52.881258 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:33:52.881468 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:33:52.881648 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:33:52.882953 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:52.884016 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:52.885665 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:52.886256 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:52.886410 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:33:52.891273 (+ 4863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:33:52.891289 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:33:52.891305 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:33:52.891356 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:33:52.894493 (+ 3137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:33:52.895265 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:33:52.895273 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:33:52.895275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:33:52.895428 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:33:52.895804 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:33:52.895807 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:33:52.895809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:33:52.896161 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:33:52.896380 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0926 16:33:52.896670 (+ 290us) server_negotiation.cc:299] Negotiation successful
0926 16:33:52.896969 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":321,"threads_started":1}
I20260926 16:33:52.898873 18968 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:33:52.999480 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 18836
I20260926 16:33:53.014933 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 18979
Sep 26 16:33:53 dist-test-slave-khd3 krb5kdc[18141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.3@KRBTEST.COM
I20260926 16:33:53.032184 19116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:33:53.015986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34256 (local address 127.6.33.3:45599)
0926 16:33:53.016815 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0926 16:33:53.016831 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 16:33:53.016860 (+ 29us) server_negotiation.cc:407] Connection header received
0926 16:33:53.016920 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:33:53.016928 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:33:53.017224 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:33:53.017352 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:33:53.018613 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:53.019676 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:53.021174 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:53.021788 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:53.021909 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:33:53.026537 (+ 4628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:33:53.026551 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:33:53.026567 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:33:53.026605 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:33:53.029479 (+ 2874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:33:53.030151 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:33:53.030158 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:33:53.030160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:33:53.030288 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:33:53.030602 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:33:53.030605 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:33:53.030607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:33:53.030928 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:33:53.031164 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0926 16:33:53.031414 (+ 250us) server_negotiation.cc:299] Negotiation successful
0926 16:33:53.031702 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":459,"threads_started":1}
I20260926 16:33:53.033576 19102 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:33:53.117698 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 18979
I20260926 16:33:53.133033 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 18585
I20260926 16:33:53.134387 18647 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:33:53.242427 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 18585
I20260926 16:33:53.263204 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:46825
--webserver_interface=127.6.33.62
--webserver_port=43693
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:36459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.33.62:46825
--ranger_config_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.33.20:46791/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.33.20:46791/kms with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:33:53.599751 19119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:33:53.600066 19119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:33:53.600209 19119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:33:53.607683 19119 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:33:53.607800 19119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:33:53.607898 19119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:33:53.607976 19119 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:33:53.608045 19119 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:33:53.618607 19119 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:36459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.33.20:46791/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.6.33.62:46825
--ranger_config_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:46825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.33.62
--webserver_port=43693
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:33:53.619988 19119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:33:53.621402 19119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:33:53.635957 19125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:33:53.636027 19124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:33:53.635919 19127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:33:53.636850 19119 server_base.cc:1034] running on GCE node
I20260926 16:33:53.637688 19119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:33:53.639181 19119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:33:53.640419 19119 hybrid_clock.cc:648] HybridClock initialized: now 1790440433640358 us; error 58 us; skew 500 ppm
I20260926 16:33:53.640856 19119 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:33:53 dist-test-slave-khd3 krb5kdc[18141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440433, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:33:53.644050 19119 init.cc:377] Logged in from keytab as kudu/127.6.33.62@KRBTEST.COM (short username kudu)
I20260926 16:33:53.645619 19119 webserver.cc:460] Webserver started at http://127.6.33.62:43693/ using document root <none> and password file <none>
I20260926 16:33:53.646301 19119 fs_manager.cc:359] Metadata directory not provided
I20260926 16:33:53.646450 19119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 16:33:53 dist-test-slave-khd3 krb5kdc[18141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440433, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.62@KRBTEST.COM for HTTP/127.6.33.20@KRBTEST.COM
I20260926 16:33:53.699450 19119 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 16:33:53.703011 19135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:33:53.704895 19119 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260926 16:33:53.705132 19119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/master-0/wal
uuid: "8e5a8d2f6bda4ca48afeac012546a6aa"
format_stamp: "Formatted at 2026-09-26 16:33:49 on dist-test-slave-khd3"
server_key: "d380bcd9d977c4c136c776ccf7244cd5"
server_key_iv: "d17ad1749380495726aa1b87976f44c9"
server_key_version: "kuduclusterkey@0"
I20260926 16:33:53.706241 19119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:33:53.719359 19119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:33:53.723913 19119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:33:53.724236 19119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:33:53.745260 19119 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.62:46825
I20260926 16:33:53.745297 19186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.62:46825 every 8 connection(s)
I20260926 16:33:53.746879 19119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
I20260926 16:33:53.747674 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 19119
I20260926 16:33:53.748593 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.33.20:46791/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.1:41029
--local_ip_for_outbound_sockets=127.6.33.1
--tserver_master_addrs=127.6.33.62:46825
--webserver_port=35491
--webserver_interface=127.6.33.1
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab
--principal=kudu/127.6.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:36459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.33.20:46791/kms with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:33:53.758464 19187 sys_catalog.cc:263] Verifying existing consensus state
I20260926 16:33:53.764540 19187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e5a8d2f6bda4ca48afeac012546a6aa: Bootstrap starting.
I20260926 16:33:53.788301 19187 log.cc:824] T 00000000000000000000000000000000 P 8e5a8d2f6bda4ca48afeac012546a6aa: Log is configured to *not* fsync() on all Append() calls
I20260926 16:33:53.798177 19187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e5a8d2f6bda4ca48afeac012546a6aa: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 16:33:53.798764 19187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e5a8d2f6bda4ca48afeac012546a6aa: Bootstrap complete.
I20260926 16:33:53.808903 19187 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e5a8d2f6bda4ca48afeac012546a6aa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e5a8d2f6bda4ca48afeac012546a6aa" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 46825 } }
I20260926 16:33:53.810000 19187 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e5a8d2f6bda4ca48afeac012546a6aa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e5a8d2f6bda4ca48afeac012546a6aa, State: Initialized, Role: FOLLOWER
I20260926 16:33:53.810846 19187 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e5a8d2f6bda4ca48afeac012546a6aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e5a8d2f6bda4ca48afeac012546a6aa" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 46825 } }
I20260926 16:33:53.811123 19187 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e5a8d2f6bda4ca48afeac012546a6aa [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:33:53.811319 19187 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e5a8d2f6bda4ca48afeac012546a6aa [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:33:53.811584 19187 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e5a8d2f6bda4ca48afeac012546a6aa [term 1 FOLLOWER]: Advancing to term 2
I20260926 16:33:53.815950 19187 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e5a8d2f6bda4ca48afeac012546a6aa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e5a8d2f6bda4ca48afeac012546a6aa" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 46825 } }
I20260926 16:33:53.816514 19187 leader_election.cc:302] T 00000000000000000000000000000000 P 8e5a8d2f6bda4ca48afeac012546a6aa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e5a8d2f6bda4ca48afeac012546a6aa; no voters:
I20260926 16:33:53.817269 19187 leader_election.cc:288] T 00000000000000000000000000000000 P 8e5a8d2f6bda4ca48afeac012546a6aa [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 16:33:53.817446 19191 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e5a8d2f6bda4ca48afeac012546a6aa [term 2 FOLLOWER]: Leader election won for term 2
I20260926 16:33:53.818747 19191 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e5a8d2f6bda4ca48afeac012546a6aa [term 2 LEADER]: Becoming Leader. State: Replica: 8e5a8d2f6bda4ca48afeac012546a6aa, State: Running, Role: LEADER
I20260926 16:33:53.819376 19191 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e5a8d2f6bda4ca48afeac012546a6aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e5a8d2f6bda4ca48afeac012546a6aa" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 46825 } }
I20260926 16:33:53.819661 19187 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e5a8d2f6bda4ca48afeac012546a6aa [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:33:53.823153 19192 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e5a8d2f6bda4ca48afeac012546a6aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8e5a8d2f6bda4ca48afeac012546a6aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e5a8d2f6bda4ca48afeac012546a6aa" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 46825 } } }
I20260926 16:33:53.823575 19192 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e5a8d2f6bda4ca48afeac012546a6aa [sys.catalog]: This master's current role is: LEADER
I20260926 16:33:53.824247 19193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e5a8d2f6bda4ca48afeac012546a6aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e5a8d2f6bda4ca48afeac012546a6aa. Latest consensus state: current_term: 2 leader_uuid: "8e5a8d2f6bda4ca48afeac012546a6aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e5a8d2f6bda4ca48afeac012546a6aa" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 46825 } } }
I20260926 16:33:53.824546 19193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e5a8d2f6bda4ca48afeac012546a6aa [sys.catalog]: This master's current role is: LEADER
I20260926 16:33:53.833132 19199 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:33:53.839537 19199 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:33:53.841377 19199 catalog_manager.cc:1259] Loaded cluster ID: af54450b8ff749909bea7b8754d36b6f
I20260926 16:33:53.841619 19199 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:33:53.845678 19199 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:33:53.847522 19199 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8e5a8d2f6bda4ca48afeac012546a6aa: Loaded TSK: 0
I20260926 16:33:53.849138 19199 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:33:53.852676 19187 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260926 16:33:54.181468 19189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:33:54.181985 19189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:33:54.182199 19189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:33:54.192673 19189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:33:54.192898 19189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:33:54.193167 19189 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.1
I20260926 16:33:54.209561 19189 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:36459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.33.20:46791/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab
--principal=kudu/127.6.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.1:41029
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.33.1
--webserver_port=35491
--tserver_master_addrs=127.6.33.62:46825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.1
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:33:54.211874 19189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:33:54.213799 19189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:33:54.254910 19226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:33:54.257150 19225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:33:54.259476 19189 server_base.cc:1034] running on GCE node
W20260926 16:33:54.259729 19229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:33:54.260586 19189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:33:54.261983 19189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:33:54.265738 19189 hybrid_clock.cc:648] HybridClock initialized: now 1790440434264324 us; error 1407 us; skew 500 ppm
I20260926 16:33:54.266254 19189 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:33:54 dist-test-slave-khd3 krb5kdc[18141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440434, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:33:54.270970 19189 init.cc:377] Logged in from keytab as kudu/127.6.33.1@KRBTEST.COM (short username kudu)
I20260926 16:33:54.282042 19189 webserver.cc:460] Webserver started at http://127.6.33.1:35491/ using document root <none> and password file <none>
I20260926 16:33:54.283221 19189 fs_manager.cc:359] Metadata directory not provided
I20260926 16:33:54.283485 19189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 16:33:54 dist-test-slave-khd3 krb5kdc[18141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440434, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.1@KRBTEST.COM for HTTP/127.6.33.20@KRBTEST.COM
I20260926 16:33:54.324069 19189 fs_manager.cc:709] Time spent opening directory manager: real 0.016s user 0.011s sys 0.004s
I20260926 16:33:54.331585 19236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:33:54.333369 19189 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.006s
I20260926 16:33:54.333642 19189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-0/wal
uuid: "79be1661dfbb4578b30084cc701559de"
format_stamp: "Formatted at 2026-09-26 16:33:51 on dist-test-slave-khd3"
server_key: "6736a222fa8fa42f365494c416527ef7"
server_key_iv: "9230c8e7e20ec2eee7f432815da84129"
server_key_version: "kuduclusterkey@0"
I20260926 16:33:54.334967 19189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:33:54.360960 19189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:33:54.365744 19189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:33:54.366130 19189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:33:54.397266 19189 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:33:54.409942 19189 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:33:54.410151 19189 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260926 16:33:54.410315 19189 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:33:54.410425 19189 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:33:54.487900 19189 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.1:41029
I20260926 16:33:54.487988 19349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.1:41029 every 8 connection(s)
I20260926 16:33:54.489898 19189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
I20260926 16:33:54.493396 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 19189
I20260926 16:33:54.494202 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.33.20:46791/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.2:46627
--local_ip_for_outbound_sockets=127.6.33.2
--tserver_master_addrs=127.6.33.62:46825
--webserver_port=38759
--webserver_interface=127.6.33.2
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab
--principal=kudu/127.6.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:36459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.33.20:46791/kms with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 16:33:54 dist-test-slave-khd3 krb5kdc[18141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440434, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.1@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:33:54.550698 19353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:33:54.498133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:46825 (local address 127.6.33.1:37641)
0926 16:33:54.515651 (+ 17518us) negotiation.cc:107] Waiting for socket to connect
0926 16:33:54.515715 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0926 16:33:54.519385 (+ 3670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:33:54.525171 (+ 5786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:33:54.525183 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:33:54.525840 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:33:54.526669 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:33:54.526696 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:33:54.529598 (+ 2902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:33:54.529604 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:33:54.531340 (+ 1736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:33:54.531353 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:33:54.532498 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:33:54.532506 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:33:54.532847 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:33:54.535527 (+ 2680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:33:54.535556 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:33:54.538506 (+ 2950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:33:54.544439 (+ 5933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:33:54.544451 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:33:54.544471 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:33:54.545250 (+ 779us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:33:54.545896 (+ 646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:33:54.545904 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:33:54.545910 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:33:54.546236 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:33:54.547553 (+ 1317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:33:54.547566 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:33:54.548080 (+ 514us) client_negotiation.cc:769] Sending connection context
0926 16:33:54.548438 (+ 358us) client_negotiation.cc:240] Negotiation successful
0926 16:33:54.549142 (+ 704us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17146,"thread_start_us":17657,"threads_started":1}
I20260926 16:33:54.552707 19352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:33:54.493434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:37641 (local address 127.6.33.62:46825)
0926 16:33:54.508645 (+ 15211us) server_negotiation.cc:206] Beginning negotiation
0926 16:33:54.508666 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 16:33:54.515891 (+ 7225us) server_negotiation.cc:407] Connection header received
0926 16:33:54.520186 (+ 4295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:33:54.520208 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:33:54.520747 (+ 539us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:33:54.521238 (+ 491us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:33:54.526957 (+ 5719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:54.529144 (+ 2187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:54.531523 (+ 2379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:54.532243 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:54.532460 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:33:54.538928 (+ 6468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:33:54.538944 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:33:54.538986 (+ 42us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:33:54.539056 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:33:54.544184 (+ 5128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:33:54.545463 (+ 1279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:33:54.545477 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:33:54.545487 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:33:54.545731 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:33:54.546408 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:33:54.546413 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:33:54.546419 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:33:54.547219 (+ 800us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:33:54.547455 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0926 16:33:54.550993 (+ 3538us) server_negotiation.cc:299] Negotiation successful
0926 16:33:54.551486 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14884,"thread_start_us":15155,"threads_started":1}
I20260926 16:33:54.559377 19350 heartbeater.cc:347] Connected to a master server at 127.6.33.62:46825
I20260926 16:33:54.559727 19350 heartbeater.cc:464] Registering TS with master...
I20260926 16:33:54.560628 19350 heartbeater.cc:511] Master 127.6.33.62:46825 requested a full tablet report, sending...
W20260926 16:33:54.570061 19350 heartbeater.cc:644] Failed to heartbeat to 127.6.33.62:46825 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 16:33:54.570333 19350 heartbeater.cc:464] Registering TS with master...
I20260926 16:33:54.570681 19350 heartbeater.cc:511] Master 127.6.33.62:46825 requested a full tablet report, sending...
I20260926 16:33:54.572103 19350 heartbeater.cc:464] Registering TS with master...
I20260926 16:33:54.572408 19350 heartbeater.cc:511] Master 127.6.33.62:46825 requested a full tablet report, sending...
W20260926 16:33:54.573709 19350 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 16:33:54.939651 19354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:33:54.939985 19354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:33:54.940152 19354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:33:54.947669 19354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:33:54.947834 19354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:33:54.948014 19354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.2
I20260926 16:33:54.958389 19354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:36459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.33.20:46791/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab
--principal=kudu/127.6.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.2:46627
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.33.2
--webserver_port=38759
--tserver_master_addrs=127.6.33.62:46825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.2
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:33:54.959815 19354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:33:54.961236 19354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:33:54.976509 19366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:33:54.976475 19367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:33:54.977046 19369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:33:54.977631 19354 server_base.cc:1034] running on GCE node
I20260926 16:33:54.978422 19354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:33:54.979550 19354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:33:54.980849 19354 hybrid_clock.cc:648] HybridClock initialized: now 1790440434980773 us; error 57 us; skew 500 ppm
I20260926 16:33:54.981263 19354 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:33:54 dist-test-slave-khd3 krb5kdc[18141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440434, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:33:54.984681 19354 init.cc:377] Logged in from keytab as kudu/127.6.33.2@KRBTEST.COM (short username kudu)
I20260926 16:33:54.993173 19354 webserver.cc:460] Webserver started at http://127.6.33.2:38759/ using document root <none> and password file <none>
I20260926 16:33:54.994015 19354 fs_manager.cc:359] Metadata directory not provided
I20260926 16:33:54.994230 19354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 16:33:54 dist-test-slave-khd3 krb5kdc[18141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440434, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.2@KRBTEST.COM for HTTP/127.6.33.20@KRBTEST.COM
I20260926 16:33:55.023288 19354 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.000s sys 0.006s
I20260926 16:33:55.026816 19379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:33:55.028486 19354 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 16:33:55.028682 19354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-1/wal
uuid: "af7963a2edd54c2a9f3e58cad99440cc"
format_stamp: "Formatted at 2026-09-26 16:33:51 on dist-test-slave-khd3"
server_key: "4649cbe4a65c0218e9f7acec40e1dafd"
server_key_iv: "2c0b9b699b880ae908cac63bea960b1c"
server_key_version: "kuduclusterkey@0"
I20260926 16:33:55.029814 19354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:33:55.071606 19354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:33:55.076844 19354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:33:55.077206 19354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:33:55.085125 19354 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:33:55.087006 19354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:33:55.087178 19354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:33:55.087332 19354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:33:55.087463 19354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:33:55.182263 19354 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.2:46627
I20260926 16:33:55.182328 19493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.2:46627 every 8 connection(s)
I20260926 16:33:55.184257 19354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
I20260926 16:33:55.189096 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 19354
I20260926 16:33:55.190089 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.33.20:46791/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.3:45599
--local_ip_for_outbound_sockets=127.6.33.3
--tserver_master_addrs=127.6.33.62:46825
--webserver_port=43169
--webserver_interface=127.6.33.3
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab
--principal=kudu/127.6.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:36459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.33.20:46791/kms with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 16:33:55 dist-test-slave-khd3 krb5kdc[18141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440434, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.2@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
Sep 26 16:33:55 dist-test-slave-khd3 krb5kdc[18141](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790440435, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:33:55.244059 19497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:33:55.188334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:57867 (local address 127.6.33.62:46825)
0926 16:33:55.210057 (+ 21723us) server_negotiation.cc:206] Beginning negotiation
0926 16:33:55.210065 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:33:55.212843 (+ 2778us) server_negotiation.cc:407] Connection header received
0926 16:33:55.219576 (+ 6733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:33:55.219583 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:33:55.219775 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:33:55.219869 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:33:55.223727 (+ 3858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:55.225021 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:55.227667 (+ 2646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:55.228432 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:55.228540 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:33:55.234003 (+ 5463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:33:55.234015 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:33:55.234024 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:33:55.234088 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:33:55.239024 (+ 4936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:33:55.240600 (+ 1576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:33:55.240609 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:33:55.240614 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:33:55.240875 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:33:55.241576 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:33:55.241585 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:33:55.241588 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:33:55.242061 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:33:55.242216 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 16:33:55.243421 (+ 1205us) server_negotiation.cc:299] Negotiation successful
0926 16:33:55.243669 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21565,"thread_start_us":21741,"threads_started":1}
I20260926 16:33:55.256882 19496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:33:55.187755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:46825 (local address 127.6.33.2:57867)
0926 16:33:55.210847 (+ 23092us) negotiation.cc:107] Waiting for socket to connect
0926 16:33:55.210921 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0926 16:33:55.219116 (+ 8195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:33:55.220538 (+ 1422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:33:55.220552 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:33:55.222116 (+ 1564us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:33:55.223050 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:33:55.223079 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:33:55.225232 (+ 2153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:33:55.225238 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:33:55.227477 (+ 2239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:33:55.227496 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:33:55.228963 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:33:55.228970 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:33:55.229263 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:33:55.230152 (+ 889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:33:55.230192 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:33:55.233590 (+ 3398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:33:55.239278 (+ 5688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:33:55.239291 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:33:55.239405 (+ 114us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:33:55.240406 (+ 1001us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:33:55.241089 (+ 683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:33:55.241094 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:33:55.241099 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:33:55.241422 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:33:55.242340 (+ 918us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:33:55.242351 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:33:55.242909 (+ 558us) client_negotiation.cc:769] Sending connection context
0926 16:33:55.245339 (+ 2430us) client_negotiation.cc:240] Negotiation successful
0926 16:33:55.246003 (+ 664us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22712,"thread_start_us":29171,"threads_started":1}
I20260926 16:33:55.259886 19494 heartbeater.cc:347] Connected to a master server at 127.6.33.62:46825
I20260926 16:33:55.260233 19494 heartbeater.cc:464] Registering TS with master...
I20260926 16:33:55.261140 19494 heartbeater.cc:511] Master 127.6.33.62:46825 requested a full tablet report, sending...
I20260926 16:33:55.263244 19152 ts_manager.cc:194] Registered new tserver with Master: af7963a2edd54c2a9f3e58cad99440cc (127.6.33.2:46627)
I20260926 16:33:55.265676 19152 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.33.2@KRBTEST.COM'} at 127.6.33.2:57867
I20260926 16:33:55.573850 19350 heartbeater.cc:464] Registering TS with master...
I20260926 16:33:55.574317 19350 heartbeater.cc:511] Master 127.6.33.62:46825 requested a full tablet report, sending...
I20260926 16:33:55.575807 19152 ts_manager.cc:194] Registered new tserver with Master: 79be1661dfbb4578b30084cc701559de (127.6.33.1:41029)
I20260926 16:33:55.577188 19152 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.33.1@KRBTEST.COM'} at 127.6.33.1:37641
Sep 26 16:33:55 dist-test-slave-khd3 krb5kdc[18141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440435, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.62@KRBTEST.COM for HTTP/127.6.33.20@KRBTEST.COM
W20260926 16:33:55.664076 19498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:33:55.664412 19498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:33:55.664597 19498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:33:55.675740 19498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:33:55.675961 19498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:33:55.676184 19498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.3
I20260926 16:33:55.691412 19498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:36459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.33.20:46791/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab
--principal=kudu/127.6.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.3:45599
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.33.3
--webserver_port=43169
--tserver_master_addrs=127.6.33.62:46825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.3
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:33:55.693470 19498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:33:55.695289 19498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:33:55.723838 19504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:33:55.727923 19505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:33:55.733006 19508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:33:55.741321 19498 server_base.cc:1034] running on GCE node
I20260926 16:33:55.742357 19498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:33:55.759651 19498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:33:55.761026 19498 hybrid_clock.cc:648] HybridClock initialized: now 1790440435760940 us; error 89 us; skew 500 ppm
I20260926 16:33:55.761508 19498 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:33:55 dist-test-slave-khd3 krb5kdc[18141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440435, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:33:55.765883 19498 init.cc:377] Logged in from keytab as kudu/127.6.33.3@KRBTEST.COM (short username kudu)
I20260926 16:33:55.770689 19498 webserver.cc:460] Webserver started at http://127.6.33.3:43169/ using document root <none> and password file <none>
I20260926 16:33:55.771785 19498 fs_manager.cc:359] Metadata directory not provided
I20260926 16:33:55.771999 19498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 16:33:55 dist-test-slave-khd3 krb5kdc[18141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440435, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.3@KRBTEST.COM for HTTP/127.6.33.20@KRBTEST.COM
I20260926 16:33:55.816342 19498 fs_manager.cc:709] Time spent opening directory manager: real 0.020s user 0.005s sys 0.011s
I20260926 16:33:55.820555 19515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:33:55.822726 19498 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 16:33:55.823020 19498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-2/wal
uuid: "df9078e09f92434eb37ede4920ffb6f9"
format_stamp: "Formatted at 2026-09-26 16:33:52 on dist-test-slave-khd3"
server_key: "0f92b755c3213b8699c9d157f328e8ec"
server_key_iv: "4f831cbadec5294fa1d78def0374cf51"
server_key_version: "kuduclusterkey@0"
I20260926 16:33:55.824599 19498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:33:55.852649 19498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:33:55.858345 19498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:33:55.858752 19498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:33:55.860114 19498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:33:55.862093 19498 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:33:55.862258 19498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260926 16:33:55.862417 19498 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:33:55.862560 19498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:33:55.911649 19498 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.3:45599
I20260926 16:33:55.911705 19627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.3:45599 every 8 connection(s)
I20260926 16:33:55.913717 19498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
I20260926 16:33:55.923671 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 19498
Sep 26 16:33:55 dist-test-slave-khd3 krb5kdc[18141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440435, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.3@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:33:55.969666 19631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:33:55.918242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.3:40457 (local address 127.6.33.62:46825)
0926 16:33:55.941045 (+ 22803us) server_negotiation.cc:206] Beginning negotiation
0926 16:33:55.941051 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:33:55.944442 (+ 3391us) server_negotiation.cc:407] Connection header received
0926 16:33:55.944518 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:33:55.944523 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:33:55.944612 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:33:55.944706 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:33:55.946876 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:55.947995 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:55.951509 (+ 3514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:55.952129 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:55.952224 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:33:55.958904 (+ 6680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:33:55.958910 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:33:55.958915 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:33:55.958961 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:33:55.962070 (+ 3109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:33:55.964435 (+ 2365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:33:55.964441 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:33:55.964444 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:33:55.964589 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:33:55.966529 (+ 1940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:33:55.966534 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:33:55.966537 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:33:55.966879 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:33:55.967006 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 16:33:55.969034 (+ 2028us) server_negotiation.cc:299] Negotiation successful
0926 16:33:55.969272 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22685,"thread_start_us":22557,"threads_started":1}
I20260926 16:33:55.973358 19633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:33:55.941004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45836 (local address 127.6.33.62:46825)
0926 16:33:55.950988 (+ 9984us) server_negotiation.cc:206] Beginning negotiation
0926 16:33:55.950996 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:33:55.951019 (+ 23us) server_negotiation.cc:407] Connection header received
0926 16:33:55.951075 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:33:55.951080 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:33:55.951160 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:33:55.951267 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:33:55.953578 (+ 2311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:55.954675 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:55.957074 (+ 2399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:55.957703 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:55.957819 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:33:55.962278 (+ 4459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:33:55.962283 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:33:55.962287 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:33:55.962318 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:33:55.965937 (+ 3619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:33:55.967057 (+ 1120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:33:55.967061 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:33:55.967064 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:33:55.967185 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:33:55.968213 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:33:55.968218 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:33:55.968222 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:33:55.968589 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:33:55.968867 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0926 16:33:55.972791 (+ 3924us) server_negotiation.cc:299] Negotiation successful
0926 16:33:55.973033 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9842,"thread_start_us":9706,"threads_started":1}
I20260926 16:33:55.974782 19630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:33:55.917636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:46825 (local address 127.6.33.3:40457)
0926 16:33:55.941649 (+ 24013us) negotiation.cc:107] Waiting for socket to connect
0926 16:33:55.941717 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0926 16:33:55.944010 (+ 2293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:33:55.945174 (+ 1164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:33:55.945186 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:33:55.945794 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:33:55.946652 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:33:55.946679 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:33:55.948223 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:33:55.948228 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:33:55.950469 (+ 2241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:33:55.950485 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:33:55.952586 (+ 2101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:33:55.952592 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:33:55.952914 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:33:55.955113 (+ 2199us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:33:55.955143 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:33:55.958649 (+ 3506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:33:55.962925 (+ 4276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:33:55.962936 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:33:55.962956 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:33:55.963541 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:33:55.966103 (+ 2562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:33:55.966107 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:33:55.966111 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:33:55.966391 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:33:55.967367 (+ 976us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:33:55.967376 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:33:55.967857 (+ 481us) client_negotiation.cc:769] Sending connection context
0926 16:33:55.970166 (+ 2309us) client_negotiation.cc:240] Negotiation successful
0926 16:33:55.971397 (+ 1231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23665,"thread_start_us":23454,"threads_started":1}
I20260926 16:33:55.976686 19628 heartbeater.cc:347] Connected to a master server at 127.6.33.62:46825
I20260926 16:33:55.977058 19628 heartbeater.cc:464] Registering TS with master...
I20260926 16:33:55.977972 19628 heartbeater.cc:511] Master 127.6.33.62:46825 requested a full tablet report, sending...
I20260926 16:33:55.979971 19152 ts_manager.cc:194] Registered new tserver with Master: df9078e09f92434eb37ede4920ffb6f9 (127.6.33.3:45599)
I20260926 16:33:55.981083 19152 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.33.3@KRBTEST.COM'} at 127.6.33.3:40457
I20260926 16:33:55.992236 6276 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 16:33:55 dist-test-slave-khd3 krb5kdc[18141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440435, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.62@KRBTEST.COM for HTTP/127.6.33.20@KRBTEST.COM
I20260926 16:33:56.019003 19633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:33:55.998901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45846 (local address 127.6.33.62:46825)
0926 16:33:55.999134 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 16:33:55.999141 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:33:56.001607 (+ 2466us) server_negotiation.cc:407] Connection header received
0926 16:33:56.001872 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:33:56.001878 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:33:56.001978 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:33:56.002056 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:33:56.004348 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:56.005403 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:56.007482 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:56.008206 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:56.008378 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:33:56.012218 (+ 3840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:33:56.012224 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:33:56.012228 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:33:56.012274 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:33:56.015638 (+ 3364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:33:56.016515 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:33:56.016522 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:33:56.016526 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:33:56.016676 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:33:56.017126 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:33:56.017131 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:33:56.017134 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:33:56.017494 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:33:56.017703 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0926 16:33:56.018286 (+ 583us) server_negotiation.cc:299] Negotiation successful
0926 16:33:56.018685 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260926 16:33:56.027292 19152 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45846:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:33:56.039043 19152 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:33:56.082875 19644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:33:56.065560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:41029 (local address 127.0.0.1:49690)
0926 16:33:56.066462 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0926 16:33:56.066512 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 16:33:56.066710 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:33:56.072809 (+ 6099us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:33:56.072821 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:33:56.072866 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:33:56.073372 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:33:56.073390 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:33:56.076032 (+ 2642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:33:56.076040 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:33:56.079105 (+ 3065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:33:56.079125 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:33:56.081625 (+ 2500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:33:56.081634 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:33:56.081916 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:33:56.081992 (+ 76us) client_negotiation.cc:769] Sending connection context
0926 16:33:56.082132 (+ 140us) client_negotiation.cc:240] Negotiation successful
0926 16:33:56.082388 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":1064,"threads_started":1}
I20260926 16:33:56.084878 19645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:33:56.066416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49690 (local address 127.6.33.1:41029)
0926 16:33:56.071959 (+ 5543us) server_negotiation.cc:206] Beginning negotiation
0926 16:33:56.071973 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 16:33:56.072017 (+ 44us) server_negotiation.cc:407] Connection header received
0926 16:33:56.072148 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:33:56.072160 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:33:56.072491 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:33:56.073663 (+ 1172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:33:56.073936 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:56.075849 (+ 1913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:56.079360 (+ 3511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:56.081330 (+ 1970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:56.083828 (+ 2498us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:33:56.084137 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0926 16:33:56.084229 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 16:33:56.084451 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5373,"thread_start_us":5256,"threads_started":1}
I20260926 16:33:56.092999 19649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:33:56.071479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34260 (local address 127.6.33.3:45599)
0926 16:33:56.078116 (+ 6637us) server_negotiation.cc:206] Beginning negotiation
0926 16:33:56.078125 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:33:56.083599 (+ 5474us) server_negotiation.cc:407] Connection header received
0926 16:33:56.083787 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:33:56.083796 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:33:56.084087 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:33:56.084303 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:33:56.085041 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:56.086632 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:56.089367 (+ 2735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:56.091292 (+ 1925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:56.091449 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:33:56.091788 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0926 16:33:56.092371 (+ 583us) server_negotiation.cc:299] Negotiation successful
0926 16:33:56.092557 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6472,"thread_start_us":6300,"threads_started":1}
I20260926 16:33:56.094014 19644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:33:56.070824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.3:45599 (local address 127.0.0.1:34260)
0926 16:33:56.083469 (+ 12645us) negotiation.cc:107] Waiting for socket to connect
0926 16:33:56.083493 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 16:33:56.083630 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:33:56.084638 (+ 1008us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:33:56.084643 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:33:56.084667 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:33:56.084845 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:33:56.084856 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:33:56.086788 (+ 1932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:33:56.086797 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:33:56.089163 (+ 2366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:33:56.089181 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:33:56.091962 (+ 2781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:33:56.091970 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:33:56.092155 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:33:56.092196 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 16:33:56.093477 (+ 1281us) client_negotiation.cc:240] Negotiation successful
0926 16:33:56.093703 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12500,"thread_start_us":6136,"threads_started":1}
I20260926 16:33:56.098547 19563 tablet_service.cc:1467] Processing CreateTablet for tablet 176d6131c0ea4ca3876b252d877de22c (DEFAULT_TABLE table=test-table [id=c14bc704ec2d4be6b5c1a6a6f2613188]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:33:56.100171 19563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 176d6131c0ea4ca3876b252d877de22c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:33:56.100453 19647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:33:56.070359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.2:46627 (local address 127.0.0.1:46902)
0926 16:33:56.077277 (+ 6918us) negotiation.cc:107] Waiting for socket to connect
0926 16:33:56.077309 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 16:33:56.077479 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:33:56.093326 (+ 15847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:33:56.093332 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:33:56.093363 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:33:56.093812 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:33:56.093829 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:33:56.095782 (+ 1953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:33:56.095788 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:33:56.097785 (+ 1997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:33:56.097801 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:33:56.099689 (+ 1888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:33:56.099695 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:33:56.099858 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:33:56.099912 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 16:33:56.099999 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 16:33:56.100131 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6753,"thread_start_us":25248,"threads_started":1}
I20260926 16:33:56.099933 19284 tablet_service.cc:1467] Processing CreateTablet for tablet 176d6131c0ea4ca3876b252d877de22c (DEFAULT_TABLE table=test-table [id=c14bc704ec2d4be6b5c1a6a6f2613188]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:33:56.101373 19284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 176d6131c0ea4ca3876b252d877de22c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:33:56.101868 19648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:33:56.070876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46902 (local address 127.6.33.2:46627)
0926 16:33:56.092347 (+ 21471us) server_negotiation.cc:206] Beginning negotiation
0926 16:33:56.092356 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:33:56.092386 (+ 30us) server_negotiation.cc:407] Connection header received
0926 16:33:56.092497 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:33:56.092506 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:33:56.093101 (+ 595us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:33:56.093333 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:33:56.094082 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:56.095611 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:56.098002 (+ 2391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:56.099458 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:56.101052 (+ 1594us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:33:56.101246 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0926 16:33:56.101317 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 16:33:56.101556 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21273,"thread_start_us":21920,"threads_started":1}
I20260926 16:33:56.105361 19429 tablet_service.cc:1467] Processing CreateTablet for tablet 176d6131c0ea4ca3876b252d877de22c (DEFAULT_TABLE table=test-table [id=c14bc704ec2d4be6b5c1a6a6f2613188]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:33:56.106612 19429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 176d6131c0ea4ca3876b252d877de22c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:33:56.129511 19650 tablet_bootstrap.cc:492] T 176d6131c0ea4ca3876b252d877de22c P df9078e09f92434eb37ede4920ffb6f9: Bootstrap starting.
I20260926 16:33:56.133598 19650 tablet_bootstrap.cc:654] T 176d6131c0ea4ca3876b252d877de22c P df9078e09f92434eb37ede4920ffb6f9: Neither blocks nor log segments found. Creating new log.
I20260926 16:33:56.138198 19650 log.cc:824] T 176d6131c0ea4ca3876b252d877de22c P df9078e09f92434eb37ede4920ffb6f9: Log is configured to *not* fsync() on all Append() calls
I20260926 16:33:56.146032 19651 tablet_bootstrap.cc:492] T 176d6131c0ea4ca3876b252d877de22c P af7963a2edd54c2a9f3e58cad99440cc: Bootstrap starting.
I20260926 16:33:56.150797 19651 tablet_bootstrap.cc:654] T 176d6131c0ea4ca3876b252d877de22c P af7963a2edd54c2a9f3e58cad99440cc: Neither blocks nor log segments found. Creating new log.
I20260926 16:33:56.154074 19650 tablet_bootstrap.cc:492] T 176d6131c0ea4ca3876b252d877de22c P df9078e09f92434eb37ede4920ffb6f9: No bootstrap required, opened a new log
I20260926 16:33:56.154479 19650 ts_tablet_manager.cc:1397] T 176d6131c0ea4ca3876b252d877de22c P df9078e09f92434eb37ede4920ffb6f9: Time spent bootstrapping tablet: real 0.025s user 0.009s sys 0.000s
I20260926 16:33:56.165632 19652 tablet_bootstrap.cc:492] T 176d6131c0ea4ca3876b252d877de22c P 79be1661dfbb4578b30084cc701559de: Bootstrap starting.
I20260926 16:33:56.167440 19651 log.cc:824] T 176d6131c0ea4ca3876b252d877de22c P af7963a2edd54c2a9f3e58cad99440cc: Log is configured to *not* fsync() on all Append() calls
I20260926 16:33:56.176970 19652 tablet_bootstrap.cc:654] T 176d6131c0ea4ca3876b252d877de22c P 79be1661dfbb4578b30084cc701559de: Neither blocks nor log segments found. Creating new log.
I20260926 16:33:56.179200 19652 log.cc:824] T 176d6131c0ea4ca3876b252d877de22c P 79be1661dfbb4578b30084cc701559de: Log is configured to *not* fsync() on all Append() calls
I20260926 16:33:56.183743 19650 raft_consensus.cc:348] T 176d6131c0ea4ca3876b252d877de22c P df9078e09f92434eb37ede4920ffb6f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79be1661dfbb4578b30084cc701559de" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 41029 } } peers { permanent_uuid: "af7963a2edd54c2a9f3e58cad99440cc" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 46627 } } peers { permanent_uuid: "df9078e09f92434eb37ede4920ffb6f9" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45599 } }
I20260926 16:33:56.184242 19650 raft_consensus.cc:374] T 176d6131c0ea4ca3876b252d877de22c P df9078e09f92434eb37ede4920ffb6f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:33:56.188220 19651 tablet_bootstrap.cc:492] T 176d6131c0ea4ca3876b252d877de22c P af7963a2edd54c2a9f3e58cad99440cc: No bootstrap required, opened a new log
I20260926 16:33:56.188666 19651 ts_tablet_manager.cc:1397] T 176d6131c0ea4ca3876b252d877de22c P af7963a2edd54c2a9f3e58cad99440cc: Time spent bootstrapping tablet: real 0.043s user 0.004s sys 0.006s
I20260926 16:33:56.195910 19652 tablet_bootstrap.cc:492] T 176d6131c0ea4ca3876b252d877de22c P 79be1661dfbb4578b30084cc701559de: No bootstrap required, opened a new log
I20260926 16:33:56.196332 19652 ts_tablet_manager.cc:1397] T 176d6131c0ea4ca3876b252d877de22c P 79be1661dfbb4578b30084cc701559de: Time spent bootstrapping tablet: real 0.031s user 0.001s sys 0.009s
I20260926 16:33:56.203225 19651 raft_consensus.cc:348] T 176d6131c0ea4ca3876b252d877de22c P af7963a2edd54c2a9f3e58cad99440cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79be1661dfbb4578b30084cc701559de" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 41029 } } peers { permanent_uuid: "af7963a2edd54c2a9f3e58cad99440cc" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 46627 } } peers { permanent_uuid: "df9078e09f92434eb37ede4920ffb6f9" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45599 } }
I20260926 16:33:56.207569 19652 raft_consensus.cc:348] T 176d6131c0ea4ca3876b252d877de22c P 79be1661dfbb4578b30084cc701559de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79be1661dfbb4578b30084cc701559de" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 41029 } } peers { permanent_uuid: "af7963a2edd54c2a9f3e58cad99440cc" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 46627 } } peers { permanent_uuid: "df9078e09f92434eb37ede4920ffb6f9" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45599 } }
I20260926 16:33:56.208228 19652 raft_consensus.cc:374] T 176d6131c0ea4ca3876b252d877de22c P 79be1661dfbb4578b30084cc701559de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:33:56.208397 19652 raft_consensus.cc:729] T 176d6131c0ea4ca3876b252d877de22c P 79be1661dfbb4578b30084cc701559de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79be1661dfbb4578b30084cc701559de, State: Initialized, Role: FOLLOWER
I20260926 16:33:56.209347 19652 consensus_queue.cc:260] T 176d6131c0ea4ca3876b252d877de22c P 79be1661dfbb4578b30084cc701559de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79be1661dfbb4578b30084cc701559de" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 41029 } } peers { permanent_uuid: "af7963a2edd54c2a9f3e58cad99440cc" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 46627 } } peers { permanent_uuid: "df9078e09f92434eb37ede4920ffb6f9" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45599 } }
I20260926 16:33:56.212858 19650 raft_consensus.cc:729] T 176d6131c0ea4ca3876b252d877de22c P df9078e09f92434eb37ede4920ffb6f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df9078e09f92434eb37ede4920ffb6f9, State: Initialized, Role: FOLLOWER
I20260926 16:33:56.213732 19650 consensus_queue.cc:260] T 176d6131c0ea4ca3876b252d877de22c P df9078e09f92434eb37ede4920ffb6f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79be1661dfbb4578b30084cc701559de" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 41029 } } peers { permanent_uuid: "af7963a2edd54c2a9f3e58cad99440cc" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 46627 } } peers { permanent_uuid: "df9078e09f92434eb37ede4920ffb6f9" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45599 } }
I20260926 16:33:56.217386 19650 ts_tablet_manager.cc:1428] T 176d6131c0ea4ca3876b252d877de22c P df9078e09f92434eb37ede4920ffb6f9: Time spent starting tablet: real 0.063s user 0.012s sys 0.004s
I20260926 16:33:56.217727 19651 raft_consensus.cc:374] T 176d6131c0ea4ca3876b252d877de22c P af7963a2edd54c2a9f3e58cad99440cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:33:56.217882 19651 raft_consensus.cc:729] T 176d6131c0ea4ca3876b252d877de22c P af7963a2edd54c2a9f3e58cad99440cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af7963a2edd54c2a9f3e58cad99440cc, State: Initialized, Role: FOLLOWER
I20260926 16:33:56.218545 19652 ts_tablet_manager.cc:1428] T 176d6131c0ea4ca3876b252d877de22c P 79be1661dfbb4578b30084cc701559de: Time spent starting tablet: real 0.022s user 0.017s sys 0.003s
I20260926 16:33:56.218742 19651 consensus_queue.cc:260] T 176d6131c0ea4ca3876b252d877de22c P af7963a2edd54c2a9f3e58cad99440cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79be1661dfbb4578b30084cc701559de" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 41029 } } peers { permanent_uuid: "af7963a2edd54c2a9f3e58cad99440cc" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 46627 } } peers { permanent_uuid: "df9078e09f92434eb37ede4920ffb6f9" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45599 } }
I20260926 16:33:56.219019 19628 heartbeater.cc:503] Master 127.6.33.62:46825 was elected leader, sending a full tablet report...
I20260926 16:33:56.226259 19350 heartbeater.cc:503] Master 127.6.33.62:46825 was elected leader, sending a full tablet report...
I20260926 16:33:56.233902 19494 heartbeater.cc:503] Master 127.6.33.62:46825 was elected leader, sending a full tablet report...
I20260926 16:33:56.235199 19651 ts_tablet_manager.cc:1428] T 176d6131c0ea4ca3876b252d877de22c P af7963a2edd54c2a9f3e58cad99440cc: Time spent starting tablet: real 0.046s user 0.018s sys 0.000s
W20260926 16:33:56.263456 19351 tablet.cc:2369] T 176d6131c0ea4ca3876b252d877de22c P 79be1661dfbb4578b30084cc701559de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:33:56.308058 19658 raft_consensus.cc:482] T 176d6131c0ea4ca3876b252d877de22c P af7963a2edd54c2a9f3e58cad99440cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:33:56.308360 19658 raft_consensus.cc:504] T 176d6131c0ea4ca3876b252d877de22c P af7963a2edd54c2a9f3e58cad99440cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79be1661dfbb4578b30084cc701559de" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 41029 } } peers { permanent_uuid: "af7963a2edd54c2a9f3e58cad99440cc" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 46627 } } peers { permanent_uuid: "df9078e09f92434eb37ede4920ffb6f9" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45599 } }
I20260926 16:33:56.310456 19658 leader_election.cc:288] T 176d6131c0ea4ca3876b252d877de22c P af7963a2edd54c2a9f3e58cad99440cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79be1661dfbb4578b30084cc701559de (127.6.33.1:41029), df9078e09f92434eb37ede4920ffb6f9 (127.6.33.3:45599)
I20260926 16:33:56.324714 19645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:33:56.311839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:49545 (local address 127.6.33.1:41029)
0926 16:33:56.312030 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 16:33:56.312037 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:33:56.317420 (+ 5383us) server_negotiation.cc:407] Connection header received
0926 16:33:56.317638 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:33:56.317644 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:33:56.317732 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:33:56.317828 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:33:56.318583 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:56.319689 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:56.322345 (+ 2656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:56.323532 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:56.323634 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:33:56.323741 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 16:33:56.324303 (+ 562us) server_negotiation.cc:299] Negotiation successful
0926 16:33:56.324459 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 16:33:56.325719 19659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:33:56.311083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:41029 (local address 127.6.33.2:49545)
0926 16:33:56.317295 (+ 6212us) negotiation.cc:107] Waiting for socket to connect
0926 16:33:56.317318 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 16:33:56.317496 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:33:56.318020 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:33:56.318025 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:33:56.318054 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:33:56.318385 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:33:56.318396 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:33:56.319911 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:33:56.319916 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:33:56.322148 (+ 2232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:33:56.322167 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:33:56.323937 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:33:56.323943 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:33:56.324111 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:33:56.324144 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 16:33:56.325243 (+ 1099us) client_negotiation.cc:240] Negotiation successful
0926 16:33:56.325387 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6062,"thread_start_us":5881,"threads_started":1}
I20260926 16:33:56.327019 19304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "176d6131c0ea4ca3876b252d877de22c" candidate_uuid: "af7963a2edd54c2a9f3e58cad99440cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79be1661dfbb4578b30084cc701559de" is_pre_election: true
I20260926 16:33:56.327625 19304 raft_consensus.cc:2454] T 176d6131c0ea4ca3876b252d877de22c P 79be1661dfbb4578b30084cc701559de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af7963a2edd54c2a9f3e58cad99440cc in term 0.
I20260926 16:33:56.328533 19381 leader_election.cc:302] T 176d6131c0ea4ca3876b252d877de22c P af7963a2edd54c2a9f3e58cad99440cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79be1661dfbb4578b30084cc701559de, af7963a2edd54c2a9f3e58cad99440cc; no voters:
I20260926 16:33:56.329103 19658 raft_consensus.cc:2781] T 176d6131c0ea4ca3876b252d877de22c P af7963a2edd54c2a9f3e58cad99440cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:33:56.329241 19658 raft_consensus.cc:482] T 176d6131c0ea4ca3876b252d877de22c P af7963a2edd54c2a9f3e58cad99440cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:33:56.329329 19658 raft_consensus.cc:3037] T 176d6131c0ea4ca3876b252d877de22c P af7963a2edd54c2a9f3e58cad99440cc [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:33:56.333276 19658 raft_consensus.cc:504] T 176d6131c0ea4ca3876b252d877de22c P af7963a2edd54c2a9f3e58cad99440cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79be1661dfbb4578b30084cc701559de" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 41029 } } peers { permanent_uuid: "af7963a2edd54c2a9f3e58cad99440cc" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 46627 } } peers { permanent_uuid: "df9078e09f92434eb37ede4920ffb6f9" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45599 } }
I20260926 16:33:56.336969 19658 leader_election.cc:288] T 176d6131c0ea4ca3876b252d877de22c P af7963a2edd54c2a9f3e58cad99440cc [CANDIDATE]: Term 1 election: Requested vote from peers 79be1661dfbb4578b30084cc701559de (127.6.33.1:41029), df9078e09f92434eb37ede4920ffb6f9 (127.6.33.3:45599)
I20260926 16:33:56.337106 19304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "176d6131c0ea4ca3876b252d877de22c" candidate_uuid: "af7963a2edd54c2a9f3e58cad99440cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79be1661dfbb4578b30084cc701559de"
I20260926 16:33:56.337319 19304 raft_consensus.cc:3037] T 176d6131c0ea4ca3876b252d877de22c P 79be1661dfbb4578b30084cc701559de [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:33:56.340533 19304 raft_consensus.cc:2454] T 176d6131c0ea4ca3876b252d877de22c P 79be1661dfbb4578b30084cc701559de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af7963a2edd54c2a9f3e58cad99440cc in term 1.
I20260926 16:33:56.341077 19381 leader_election.cc:302] T 176d6131c0ea4ca3876b252d877de22c P af7963a2edd54c2a9f3e58cad99440cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79be1661dfbb4578b30084cc701559de, af7963a2edd54c2a9f3e58cad99440cc; no voters:
I20260926 16:33:56.341388 19658 raft_consensus.cc:2781] T 176d6131c0ea4ca3876b252d877de22c P af7963a2edd54c2a9f3e58cad99440cc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:33:56.341876 19649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:33:56.312888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:34959 (local address 127.6.33.3:45599)
0926 16:33:56.313059 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 16:33:56.313065 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:33:56.332462 (+ 19397us) server_negotiation.cc:407] Connection header received
0926 16:33:56.332662 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:33:56.332668 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:33:56.332747 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:33:56.332867 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:33:56.333560 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:56.334657 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:56.339025 (+ 4368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:56.340305 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:56.340437 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:33:56.340560 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 16:33:56.341464 (+ 904us) server_negotiation.cc:299] Negotiation successful
0926 16:33:56.341623 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 16:33:56.342037 19660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:33:56.312420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.3:45599 (local address 127.6.33.2:34959)
0926 16:33:56.332338 (+ 19918us) negotiation.cc:107] Waiting for socket to connect
0926 16:33:56.332360 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 16:33:56.332522 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:33:56.332989 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:33:56.332994 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:33:56.333022 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:33:56.333379 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:33:56.333391 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:33:56.334909 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:33:56.334915 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:33:56.338466 (+ 3551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:33:56.338484 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:33:56.340837 (+ 2353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:33:56.340844 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:33:56.341025 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:33:56.341066 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 16:33:56.341160 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 16:33:56.341312 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19744,"thread_start_us":20191,"threads_started":1}
I20260926 16:33:56.343477 19582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "176d6131c0ea4ca3876b252d877de22c" candidate_uuid: "af7963a2edd54c2a9f3e58cad99440cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df9078e09f92434eb37ede4920ffb6f9"
I20260926 16:33:56.343407 19583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "176d6131c0ea4ca3876b252d877de22c" candidate_uuid: "af7963a2edd54c2a9f3e58cad99440cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df9078e09f92434eb37ede4920ffb6f9" is_pre_election: true
I20260926 16:33:56.344058 19583 raft_consensus.cc:2454] T 176d6131c0ea4ca3876b252d877de22c P df9078e09f92434eb37ede4920ffb6f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af7963a2edd54c2a9f3e58cad99440cc in term 0.
I20260926 16:33:56.345543 19658 raft_consensus.cc:686] T 176d6131c0ea4ca3876b252d877de22c P af7963a2edd54c2a9f3e58cad99440cc [term 1 LEADER]: Becoming Leader. State: Replica: af7963a2edd54c2a9f3e58cad99440cc, State: Running, Role: LEADER
I20260926 16:33:56.346118 19658 consensus_queue.cc:237] T 176d6131c0ea4ca3876b252d877de22c P af7963a2edd54c2a9f3e58cad99440cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79be1661dfbb4578b30084cc701559de" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 41029 } } peers { permanent_uuid: "af7963a2edd54c2a9f3e58cad99440cc" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 46627 } } peers { permanent_uuid: "df9078e09f92434eb37ede4920ffb6f9" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45599 } }
I20260926 16:33:56.351795 19152 catalog_manager.cc:5495] T 176d6131c0ea4ca3876b252d877de22c P af7963a2edd54c2a9f3e58cad99440cc reported cstate change: term changed from 0 to 1, leader changed from <none> to af7963a2edd54c2a9f3e58cad99440cc (127.6.33.2). New cstate: current_term: 1 leader_uuid: "af7963a2edd54c2a9f3e58cad99440cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79be1661dfbb4578b30084cc701559de" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 41029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af7963a2edd54c2a9f3e58cad99440cc" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 46627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df9078e09f92434eb37ede4920ffb6f9" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45599 } health_report { overall_health: UNKNOWN } } }
W20260926 16:33:56.421703 19629 tablet.cc:2369] T 176d6131c0ea4ca3876b252d877de22c P df9078e09f92434eb37ede4920ffb6f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:33:56.425501 19648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:33:56.419191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33704 (local address 127.6.33.2:46627)
0926 16:33:56.419523 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0926 16:33:56.419530 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:33:56.419550 (+ 20us) server_negotiation.cc:407] Connection header received
0926 16:33:56.419815 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:33:56.419819 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:33:56.419896 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:33:56.420100 (+ 204us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:33:56.420479 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:56.421306 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:56.422847 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:56.423449 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:56.423595 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:33:56.424230 (+ 635us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:33:56.424857 (+ 627us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:33:56.424977 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 16:33:56.425109 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 16:33:56.425181 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167}
I20260926 16:33:56.434479 19583 raft_consensus.cc:3037] T 176d6131c0ea4ca3876b252d877de22c P df9078e09f92434eb37ede4920ffb6f9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:33:56.434865 19304 raft_consensus.cc:1261] T 176d6131c0ea4ca3876b252d877de22c P 79be1661dfbb4578b30084cc701559de [term 1 FOLLOWER]: Refusing update from remote peer af7963a2edd54c2a9f3e58cad99440cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:33:56.435958 19658 consensus_queue.cc:1035] T 176d6131c0ea4ca3876b252d877de22c P af7963a2edd54c2a9f3e58cad99440cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "79be1661dfbb4578b30084cc701559de" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 41029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:33:56.437726 19583 raft_consensus.cc:1261] T 176d6131c0ea4ca3876b252d877de22c P df9078e09f92434eb37ede4920ffb6f9 [term 1 FOLLOWER]: Refusing update from remote peer af7963a2edd54c2a9f3e58cad99440cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:33:56.438654 19658 consensus_queue.cc:1035] T 176d6131c0ea4ca3876b252d877de22c P af7963a2edd54c2a9f3e58cad99440cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "df9078e09f92434eb37ede4920ffb6f9" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:33:56.449343 19671 mvcc.cc:205] Tried to move back new op lower bound from 7333644027624992768 to 7333644027278508032. Current Snapshot: MvccSnapshot[applied={T|T < 7333644027624992768}]
I20260926 16:33:56.450428 19674 mvcc.cc:205] Tried to move back new op lower bound from 7333644027624992768 to 7333644027278508032. Current Snapshot: MvccSnapshot[applied={T|T < 7333644027624992768}]
I20260926 16:33:56.452848 19672 mvcc.cc:205] Tried to move back new op lower bound from 7333644027624992768 to 7333644027278508032. Current Snapshot: MvccSnapshot[applied={T|T < 7333644027624992768}]
I20260926 16:33:56.465770 19152 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45846:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:33:56.466004 19152 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45846:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:33:56.469993 19152 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8e5a8d2f6bda4ca48afeac012546a6aa: Sending DeleteTablet for 3 replicas of tablet 176d6131c0ea4ca3876b252d877de22c
I20260926 16:33:56.471467 19284 tablet_service.cc:1514] Processing DeleteTablet for tablet 176d6131c0ea4ca3876b252d877de22c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:33:56 UTC) from {username='kudu', principal='kudu/127.6.33.62@KRBTEST.COM'} at 127.0.0.1:49690
I20260926 16:33:56.471479 19429 tablet_service.cc:1514] Processing DeleteTablet for tablet 176d6131c0ea4ca3876b252d877de22c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:33:56 UTC) from {username='kudu', principal='kudu/127.6.33.62@KRBTEST.COM'} at 127.0.0.1:46902
I20260926 16:33:56.471745 19563 tablet_service.cc:1514] Processing DeleteTablet for tablet 176d6131c0ea4ca3876b252d877de22c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:33:56 UTC) from {username='kudu', principal='kudu/127.6.33.62@KRBTEST.COM'} at 127.0.0.1:34260
I20260926 16:33:56.474431 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 19189
I20260926 16:33:56.485086 19681 tablet_replica.cc:330] T 176d6131c0ea4ca3876b252d877de22c P af7963a2edd54c2a9f3e58cad99440cc: stopping tablet replica
I20260926 16:33:56.485879 19681 raft_consensus.cc:2229] T 176d6131c0ea4ca3876b252d877de22c P af7963a2edd54c2a9f3e58cad99440cc [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:33:56.486956 19681 raft_consensus.cc:2258] T 176d6131c0ea4ca3876b252d877de22c P af7963a2edd54c2a9f3e58cad99440cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:33:56.488054 19680 tablet_replica.cc:330] T 176d6131c0ea4ca3876b252d877de22c P 79be1661dfbb4578b30084cc701559de: stopping tablet replica
I20260926 16:33:56.488181 19682 tablet_replica.cc:330] T 176d6131c0ea4ca3876b252d877de22c P df9078e09f92434eb37ede4920ffb6f9: stopping tablet replica
I20260926 16:33:56.488586 19682 raft_consensus.cc:2229] T 176d6131c0ea4ca3876b252d877de22c P df9078e09f92434eb37ede4920ffb6f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:33:56.488603 19680 raft_consensus.cc:2229] T 176d6131c0ea4ca3876b252d877de22c P 79be1661dfbb4578b30084cc701559de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:33:56.489152 19680 raft_consensus.cc:2258] T 176d6131c0ea4ca3876b252d877de22c P 79be1661dfbb4578b30084cc701559de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:33:56.489144 19682 raft_consensus.cc:2258] T 176d6131c0ea4ca3876b252d877de22c P df9078e09f92434eb37ede4920ffb6f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:33:56.489252 19681 ts_tablet_manager.cc:1905] T 176d6131c0ea4ca3876b252d877de22c P af7963a2edd54c2a9f3e58cad99440cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:33:56.491184 19680 ts_tablet_manager.cc:1905] T 176d6131c0ea4ca3876b252d877de22c P 79be1661dfbb4578b30084cc701559de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:33:56.491458 19682 ts_tablet_manager.cc:1905] T 176d6131c0ea4ca3876b252d877de22c P df9078e09f92434eb37ede4920ffb6f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:33:56.496618 19681 ts_tablet_manager.cc:1918] T 176d6131c0ea4ca3876b252d877de22c P af7963a2edd54c2a9f3e58cad99440cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:33:56.496912 19681 log.cc:1196] T 176d6131c0ea4ca3876b252d877de22c P af7963a2edd54c2a9f3e58cad99440cc: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-1/wal/wals/176d6131c0ea4ca3876b252d877de22c
I20260926 16:33:56.497505 19681 ts_tablet_manager.cc:1939] T 176d6131c0ea4ca3876b252d877de22c P af7963a2edd54c2a9f3e58cad99440cc: Deleting consensus metadata
I20260926 16:33:56.498476 19682 ts_tablet_manager.cc:1918] T 176d6131c0ea4ca3876b252d877de22c P df9078e09f92434eb37ede4920ffb6f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:33:56.498728 19682 log.cc:1196] T 176d6131c0ea4ca3876b252d877de22c P df9078e09f92434eb37ede4920ffb6f9: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-2/wal/wals/176d6131c0ea4ca3876b252d877de22c
I20260926 16:33:56.499321 19682 ts_tablet_manager.cc:1939] T 176d6131c0ea4ca3876b252d877de22c P df9078e09f92434eb37ede4920ffb6f9: Deleting consensus metadata
I20260926 16:33:56.499359 19137 catalog_manager.cc:4841] TS af7963a2edd54c2a9f3e58cad99440cc (127.6.33.2:46627): tablet 176d6131c0ea4ca3876b252d877de22c (table test-table [id=c14bc704ec2d4be6b5c1a6a6f2613188]) successfully deleted
I20260926 16:33:56.501238 19139 catalog_manager.cc:4841] TS df9078e09f92434eb37ede4920ffb6f9 (127.6.33.3:45599): tablet 176d6131c0ea4ca3876b252d877de22c (table test-table [id=c14bc704ec2d4be6b5c1a6a6f2613188]) successfully deleted
I20260926 16:33:56.507982 19680 ts_tablet_manager.cc:1918] T 176d6131c0ea4ca3876b252d877de22c P 79be1661dfbb4578b30084cc701559de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:33:56.508284 19680 log.cc:1196] T 176d6131c0ea4ca3876b252d877de22c P 79be1661dfbb4578b30084cc701559de: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440233850748-6276-0/minicluster-data/ts-0/wal/wals/176d6131c0ea4ca3876b252d877de22c
I20260926 16:33:56.508985 19680 ts_tablet_manager.cc:1939] T 176d6131c0ea4ca3876b252d877de22c P 79be1661dfbb4578b30084cc701559de: Deleting consensus metadata
I20260926 16:33:56.512105 19137 catalog_manager.cc:4841] TS 79be1661dfbb4578b30084cc701559de (127.6.33.1:41029): tablet 176d6131c0ea4ca3876b252d877de22c (table test-table [id=c14bc704ec2d4be6b5c1a6a6f2613188]) successfully deleted
I20260926 16:33:56.518780 19645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:33:56.475590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35370 (local address 127.6.33.1:41029)
0926 16:33:56.475815 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 16:33:56.475821 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:33:56.503729 (+ 27908us) server_negotiation.cc:407] Connection header received
0926 16:33:56.503882 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:33:56.503888 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:33:56.503992 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:33:56.504110 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:33:56.505865 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:56.507102 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:56.509149 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:56.509820 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:56.509969 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:33:56.513011 (+ 3042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:33:56.513029 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:33:56.513053 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:33:56.513093 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:33:56.515935 (+ 2842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:33:56.516612 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:33:56.516619 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:33:56.516622 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:33:56.516747 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:33:56.517176 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:33:56.517181 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:33:56.517184 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:33:56.517532 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:33:56.517868 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0926 16:33:56.518213 (+ 345us) server_negotiation.cc:299] Negotiation successful
0926 16:33:56.518422 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 16:33:56.519502 19345 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:33:56.610874 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 19189
I20260926 16:33:56.629010 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 19354
I20260926 16:33:56.645507 19648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:33:56.629860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33712 (local address 127.6.33.2:46627)
0926 16:33:56.630058 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 16:33:56.630066 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:33:56.630136 (+ 70us) server_negotiation.cc:407] Connection header received
0926 16:33:56.630339 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:33:56.630342 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:33:56.630406 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:33:56.630494 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:33:56.632188 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:56.633226 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:56.635154 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:56.635621 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:56.635762 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:33:56.639143 (+ 3381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:33:56.639165 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:33:56.639188 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:33:56.639235 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:33:56.642515 (+ 3280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:33:56.643388 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:33:56.643426 (+ 38us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:33:56.643430 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:33:56.643591 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:33:56.644018 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:33:56.644023 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:33:56.644025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:33:56.644410 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:33:56.644686 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0926 16:33:56.645014 (+ 328us) server_negotiation.cc:299] Negotiation successful
0926 16:33:56.645181 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260926 16:33:56.646322 19489 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:33:56.761878 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 19354
I20260926 16:33:56.779839 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 19498
I20260926 16:33:56.793839 19649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:33:56.780690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39596 (local address 127.6.33.3:45599)
0926 16:33:56.780886 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 16:33:56.780890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:33:56.780909 (+ 19us) server_negotiation.cc:407] Connection header received
0926 16:33:56.781147 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:33:56.781152 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:33:56.781219 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:33:56.781343 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:33:56.782691 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:56.783447 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:56.784958 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:56.785352 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:56.785455 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:33:56.787940 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:33:56.787953 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:33:56.787968 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:33:56.787998 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:33:56.790998 (+ 3000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:33:56.791714 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:33:56.791721 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:33:56.791723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:33:56.791847 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:33:56.792210 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:33:56.792213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:33:56.792215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:33:56.792584 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:33:56.792868 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0926 16:33:56.793226 (+ 358us) server_negotiation.cc:299] Negotiation successful
0926 16:33:56.793386 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 16:33:56.794579 19623 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:33:56.883841 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 19498
I20260926 16:33:56.900235 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 19119
I20260926 16:33:56.901710 19182 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:33:57.015169 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 19119
I20260926 16:33:57.037664 6276 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 16:33:57.373039 6276 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 16:33:57.373281 6276 mini_ranger.cc:66] Stopping Ranger...
I20260926 16:33:57.719019 6276 mini_ranger.cc:68] Stopped Ranger
2026-09-26 16:33:57.719 UTC [18171] LOG: received smart shutdown request
2026-09-26 16:33:57.722 UTC [18171] LOG: background worker "logical replication launcher" (PID 18180) exited with exit code 1
2026-09-26 16:33:57.723 UTC [18175] LOG: shutting down
2026-09-26 16:33:57.914 UTC [18171] LOG: database system is shut down
2026-09-26T16:33:57Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (53609 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:33:58 dist-test-slave-khd3 krb5kdc[19696](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:33:58 dist-test-slave-khd3 krb5kdc[19696](info): set up 2 sockets
Sep 26 16:33:58 dist-test-slave-khd3 krb5kdc[19696](info): commencing operation
krb5kdc: starting...
W20260926 16:33:59.146055 6276 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Sep 26 16:33:59 dist-test-slave-khd3 krb5kdc[19696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:33:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:33:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab.
Entry for principal kudu/127.6.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab.
I20260926 16:33:59.380060 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:34199
--webserver_interface=127.6.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:42373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.33.62:34199,127.6.33.61:38883,127.6.33.60:37327
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:33:59.723242 19709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:33:59.723560 19709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:33:59.723652 19709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:33:59.731312 19709 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:33:59.731426 19709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:33:59.731470 19709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:33:59.731520 19709 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:33:59.731571 19709 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:33:59.742017 19709 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:42373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.33.62:34199,127.6.33.61:38883,127.6.33.60:37327
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:34199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:33:59.743330 19709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:33:59.744630 19709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:33:59.756388 19715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:33:59.756332 19717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:33:59.756423 19714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:33:59.757071 19709 server_base.cc:1034] running on GCE node
I20260926 16:33:59.757874 19709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:33:59.759316 19709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:33:59.760576 19709 hybrid_clock.cc:648] HybridClock initialized: now 1790440439760518 us; error 64 us; skew 500 ppm
I20260926 16:33:59.761094 19709 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:33:59 dist-test-slave-khd3 krb5kdc[19696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440439, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:33:59.769995 19709 init.cc:377] Logged in from keytab as kudu/127.6.33.62@KRBTEST.COM (short username kudu)
I20260926 16:33:59.771579 19709 webserver.cc:460] Webserver started at http://127.6.33.62:36395/ using document root <none> and password file <none>
I20260926 16:33:59.772353 19709 fs_manager.cc:359] Metadata directory not provided
I20260926 16:33:59.772502 19709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:33:59.772881 19709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:33:59.775889 19709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/minicluster-data/master-0/data/instance:
uuid: "32d712db95ac4b0e9ed8b8574ef883d5"
format_stamp: "Formatted at 2026-09-26 16:33:59 on dist-test-slave-khd3"
server_key: "feeec70efbf55905b1e9f38cf9168568"
server_key_iv: "0d2de63dc020cf426b84e920ff8180e0"
server_key_version: "encryptionkey@0"
I20260926 16:33:59.776727 19709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/minicluster-data/master-0/wal/instance:
uuid: "32d712db95ac4b0e9ed8b8574ef883d5"
format_stamp: "Formatted at 2026-09-26 16:33:59 on dist-test-slave-khd3"
server_key: "feeec70efbf55905b1e9f38cf9168568"
server_key_iv: "0d2de63dc020cf426b84e920ff8180e0"
server_key_version: "encryptionkey@0"
I20260926 16:33:59.782377 19709 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 16:33:59.786324 19724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:33:59.787875 19709 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:33:59.788134 19709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/minicluster-data/master-0/wal
uuid: "32d712db95ac4b0e9ed8b8574ef883d5"
format_stamp: "Formatted at 2026-09-26 16:33:59 on dist-test-slave-khd3"
server_key: "feeec70efbf55905b1e9f38cf9168568"
server_key_iv: "0d2de63dc020cf426b84e920ff8180e0"
server_key_version: "encryptionkey@0"
I20260926 16:33:59.788353 19709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:33:59.806788 19709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:33:59.812219 19709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:33:59.812526 19709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:33:59.831439 19709 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.62:34199
I20260926 16:33:59.831460 19775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.62:34199 every 8 connection(s)
I20260926 16:33:59.832998 19709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
I20260926 16:33:59.836871 19776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:33:59.842833 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 19709
I20260926 16:33:59.843087 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/minicluster-data/master-0/wal/instance
I20260926 16:33:59.843878 6276 external_mini_cluster.cc:1428] Setting key d4c4ed24d1df732f9bc3d9a6d33caf42
I20260926 16:33:59.843801 19776 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.33.62" port: 34199 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 16:33:59 dist-test-slave-khd3 krb5kdc[19696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440439, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.62@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.33.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.61@KRBTEST.COM" created.
I20260926 16:33:59.870657 19778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:33:59.847822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:34199 (local address 127.0.0.1:58430)
0926 16:33:59.848856 (+ 1034us) negotiation.cc:107] Waiting for socket to connect
0926 16:33:59.848913 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 16:33:59.851723 (+ 2810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:33:59.852766 (+ 1043us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:33:59.852796 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:33:59.853642 (+ 846us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:33:59.854138 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:33:59.854164 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:33:59.855724 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:33:59.855732 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:33:59.857704 (+ 1972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:33:59.857722 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:33:59.858723 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:33:59.858732 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:33:59.858941 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:33:59.859675 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:33:59.859710 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:33:59.862618 (+ 2908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:33:59.866151 (+ 3533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:33:59.866164 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:33:59.866179 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:33:59.866605 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:33:59.867040 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:33:59.867085 (+ 45us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:33:59.867088 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:33:59.867242 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:33:59.868120 (+ 878us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:33:59.868127 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:33:59.868341 (+ 214us) client_negotiation.cc:769] Sending connection context
0926 16:33:59.868632 (+ 291us) client_negotiation.cc:240] Negotiation successful
0926 16:33:59.869287 (+ 655us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"mutex_wait_us":158,"thread_start_us":459,"threads_started":1}
I20260926 16:33:59.870690 19779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:33:59.847822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58430 (local address 127.6.33.62:34199)
0926 16:33:59.849932 (+ 2110us) server_negotiation.cc:206] Beginning negotiation
0926 16:33:59.849939 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:33:59.849983 (+ 44us) server_negotiation.cc:407] Connection header received
0926 16:33:59.852123 (+ 2140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:33:59.852133 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:33:59.852471 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:33:59.852712 (+ 241us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:33:59.854343 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:59.855559 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:59.857858 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:33:59.858459 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:33:59.858596 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:33:59.862865 (+ 4269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:33:59.862873 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:33:59.862886 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:33:59.862920 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:33:59.865997 (+ 3077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:33:59.866755 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:33:59.866762 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:33:59.866764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:33:59.866918 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:33:59.867357 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:33:59.867364 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:33:59.867371 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:33:59.867930 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:33:59.868148 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0926 16:33:59.868822 (+ 674us) server_negotiation.cc:299] Negotiation successful
0926 16:33:59.869149 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1914,"spinlock_wait_cycles":26496,"thread_start_us":1668,"threads_started":1}
I20260926 16:33:59.873860 19776 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.33.61" port: 38883 } has no permanent_uuid. Determining permanent_uuid...
I20260926 16:33:59.875212 19778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:33:59.874741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.61:38883 (local address 127.0.0.1:54516)
0926 16:33:59.874940 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 16:33:59.875077 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.61:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260926 16:33:59.875519 19725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.33.61:38883: connect: Connection refused (error 111)
I20260926 16:33:59.877264 19778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:33:59.876889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.61:38883 (local address 127.0.0.1:54530)
0926 16:33:59.877046 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 16:33:59.877124 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.61:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 16:33:59.877470 19776 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.33.61:38883: Network error: Client connection negotiation failed: client connection to 127.6.33.61:38883: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.61.keytab.
Entry for principal kudu/127.6.33.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.61.keytab.
I20260926 16:33:59.896134 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.61:38883
--webserver_interface=127.6.33.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.61.keytab
--principal=kudu/127.6.33.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:42373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.33.62:34199,127.6.33.61:38883,127.6.33.60:37327
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:33:59.926837 19776 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.33.61" port: 38883 } attempt: 1
I20260926 16:33:59.928045 19778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:33:59.927595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.61:38883 (local address 127.0.0.1:54536)
0926 16:33:59.927801 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 16:33:59.927894 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.61:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260926 16:33:59.929103 19778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:33:59.928707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.61:38883 (local address 127.0.0.1:54538)
0926 16:33:59.928866 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 16:33:59.928936 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.61:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260926 16:33:59.929322 19776 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.33.61:38883: Network error: Client connection negotiation failed: client connection to 127.6.33.61:38883: connect: Connection refused (error 111)
I20260926 16:33:59.997593 19776 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.33.61" port: 38883 } attempt: 2
I20260926 16:33:59.998869 19778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:33:59.998375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.61:38883 (local address 127.0.0.1:54550)
0926 16:33:59.998640 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0926 16:33:59.998728 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.61:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260926 16:33:59.999936 19778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:33:59.999564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.61:38883 (local address 127.0.0.1:54560)
0926 16:33:59.999713 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 16:33:59.999801 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.61:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 16:34:00.000201 19776 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.33.61:38883: Network error: Client connection negotiation failed: client connection to 127.6.33.61:38883: connect: Connection refused (error 111)
I20260926 16:34:00.091482 19776 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.33.61" port: 38883 } attempt: 3
I20260926 16:34:00.092810 19778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:00.092268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.61:38883 (local address 127.0.0.1:54562)
0926 16:34:00.092494 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:00.092641 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.61:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260926 16:34:00.093906 19778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:00.093555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.61:38883 (local address 127.0.0.1:54564)
0926 16:34:00.093705 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:00.093779 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.61:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 16:34:00.094202 19776 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.33.61:38883: Network error: Client connection negotiation failed: client connection to 127.6.33.61:38883: connect: Connection refused (error 111)
W20260926 16:34:00.226325 19783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:34:00.226670 19783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:34:00.226817 19783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:34:00.234272 19783 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:34:00.234421 19783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:34:00.234532 19783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:34:00.234627 19783 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:34:00.234709 19783 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:34:00.237466 19776 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.33.61" port: 38883 } attempt: 4
I20260926 16:34:00.238816 19778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:00.238245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.61:38883 (local address 127.0.0.1:54574)
0926 16:34:00.238585 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:00.238677 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.61:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187}
I20260926 16:34:00.239972 19778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:00.239546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.61:38883 (local address 127.0.0.1:54590)
0926 16:34:00.239701 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:00.239808 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.61:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 16:34:00.240170 19776 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.33.61:38883: Network error: Client connection negotiation failed: client connection to 127.6.33.61:38883: connect: Connection refused (error 111)
I20260926 16:34:00.245481 19783 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:42373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.33.62:34199,127.6.33.61:38883,127.6.33.60:37327
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.61.keytab
--principal=kudu/127.6.33.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.61:38883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.33.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:34:00.246833 19783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:34:00.248169 19783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:34:00.259975 19788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:34:00.260044 19791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:34:00.260000 19789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:34:00.260720 19783 server_base.cc:1034] running on GCE node
I20260926 16:34:00.261475 19783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:34:00.262827 19783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:34:00.264024 19783 hybrid_clock.cc:648] HybridClock initialized: now 1790440440263983 us; error 48 us; skew 500 ppm
I20260926 16:34:00.264386 19783 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:34:00 dist-test-slave-khd3 krb5kdc[19696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440440, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:34:00.267510 19783 init.cc:377] Logged in from keytab as kudu/127.6.33.61@KRBTEST.COM (short username kudu)
I20260926 16:34:00.268990 19783 webserver.cc:460] Webserver started at http://127.6.33.61:39909/ using document root <none> and password file <none>
I20260926 16:34:00.269625 19783 fs_manager.cc:359] Metadata directory not provided
I20260926 16:34:00.269721 19783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:34:00.269953 19783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:34:00.272749 19783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/minicluster-data/master-1/data/instance:
uuid: "bf27e4b5f3c9462fa5e10619b37c3a58"
format_stamp: "Formatted at 2026-09-26 16:34:00 on dist-test-slave-khd3"
server_key: "f9df86d5de8a78ccd73a90141592b8ae"
server_key_iv: "3535d39c4f057687e289518f3725484c"
server_key_version: "encryptionkey@0"
I20260926 16:34:00.273547 19783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/minicluster-data/master-1/wal/instance:
uuid: "bf27e4b5f3c9462fa5e10619b37c3a58"
format_stamp: "Formatted at 2026-09-26 16:34:00 on dist-test-slave-khd3"
server_key: "f9df86d5de8a78ccd73a90141592b8ae"
server_key_iv: "3535d39c4f057687e289518f3725484c"
server_key_version: "encryptionkey@0"
I20260926 16:34:00.278406 19783 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 16:34:00.281785 19798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:34:00.283289 19783 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:34:00.283469 19783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/minicluster-data/master-1/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/minicluster-data/master-1/wal
uuid: "bf27e4b5f3c9462fa5e10619b37c3a58"
format_stamp: "Formatted at 2026-09-26 16:34:00 on dist-test-slave-khd3"
server_key: "f9df86d5de8a78ccd73a90141592b8ae"
server_key_iv: "3535d39c4f057687e289518f3725484c"
server_key_version: "encryptionkey@0"
I20260926 16:34:00.283661 19783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:34:00.299358 19783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:34:00.303742 19783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:34:00.304039 19783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:34:00.322359 19783 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.61:38883
I20260926 16:34:00.322379 19849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.61:38883 every 8 connection(s)
I20260926 16:34:00.323702 19783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/minicluster-data/master-1/data/info.pb
I20260926 16:34:00.327467 19850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:34:00.328054 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 19783
I20260926 16:34:00.328250 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/minicluster-data/master-1/wal/instance
I20260926 16:34:00.328852 6276 external_mini_cluster.cc:1428] Setting key d3f5acfff4a052e6fd10ba3e3fb89284
I20260926 16:34:00.337843 19850 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.33.62" port: 34199 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.6.33.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.60@KRBTEST.COM" created.
Sep 26 16:34:00 dist-test-slave-khd3 krb5kdc[19696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440440, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.61@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:34:00.362407 19779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:00.340722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58440 (local address 127.6.33.62:34199)
0926 16:34:00.340940 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:00.340945 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:00.342292 (+ 1347us) server_negotiation.cc:407] Connection header received
0926 16:34:00.344821 (+ 2529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:00.344828 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:00.344923 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:00.345042 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:34:00.346617 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:00.347414 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:00.349026 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:00.349463 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:00.349630 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:00.354908 (+ 5278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:34:00.354918 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:34:00.354926 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:34:00.354986 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:34:00.358922 (+ 3936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:00.359735 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:00.359744 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:00.359750 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:00.359904 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:00.360333 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:00.360341 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:00.360347 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:00.360776 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:34:00.360943 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:00.361981 (+ 1038us) server_negotiation.cc:299] Negotiation successful
0926 16:34:00.362183 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260926 16:34:00.363890 19853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:00.341289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:34199 (local address 127.0.0.1:58440)
0926 16:34:00.342109 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:00.342153 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 16:34:00.344467 (+ 2314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:34:00.345307 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:34:00.345316 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:34:00.345844 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:34:00.346430 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:00.346447 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:00.347571 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:00.347576 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:00.348886 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:00.348898 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:00.349733 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:00.349740 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:00.349970 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:00.350699 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:34:00.350720 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:34:00.354618 (+ 3898us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:34:00.359099 (+ 4481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:34:00.359106 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:34:00.359120 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:34:00.359582 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:34:00.360024 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:34:00.360028 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:34:00.360030 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:34:00.360211 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:34:00.360961 (+ 750us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:34:00.360970 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:34:00.361516 (+ 546us) client_negotiation.cc:769] Sending connection context
0926 16:34:00.361849 (+ 333us) client_negotiation.cc:240] Negotiation successful
0926 16:34:00.362425 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":349,"threads_started":1}
I20260926 16:34:00.365734 19850 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.33.61" port: 38883 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 16:34:00 dist-test-slave-khd3 krb5kdc[19696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440440, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.61@KRBTEST.COM for kudu/127.6.33.61@KRBTEST.COM
I20260926 16:34:00.382184 19853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:00.366534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.61:38883 (local address 127.0.0.1:54602)
0926 16:34:00.366739 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:00.366760 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 16:34:00.366869 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:34:00.367877 (+ 1008us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:34:00.367889 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:34:00.368394 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:34:00.368604 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:00.368619 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:00.370035 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:00.370043 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:00.371616 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:00.371635 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:00.372589 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:00.372601 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:00.372916 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:00.373562 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:34:00.373591 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:34:00.376215 (+ 2624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:34:00.379498 (+ 3283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:34:00.379502 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:34:00.379505 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:34:00.379886 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:34:00.380338 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:34:00.380341 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:34:00.380344 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:34:00.380510 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:34:00.381271 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:34:00.381275 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:34:00.381512 (+ 237us) client_negotiation.cc:769] Sending connection context
0926 16:34:00.381646 (+ 134us) client_negotiation.cc:240] Negotiation successful
0926 16:34:00.381862 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 16:34:00.382448 19855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:00.366761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54602 (local address 127.6.33.61:38883)
0926 16:34:00.367458 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:00.367464 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:00.367489 (+ 25us) server_negotiation.cc:407] Connection header received
0926 16:34:00.367532 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:00.367537 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:00.367711 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:00.367834 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:34:00.368845 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:00.369869 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:00.371782 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:00.372421 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:00.372516 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:00.376431 (+ 3915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:34:00.376443 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:34:00.376458 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:34:00.376487 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:34:00.379350 (+ 2863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:00.380111 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:00.380118 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:00.380121 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:00.380230 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:00.380661 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:00.380665 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:00.380667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:00.381139 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:34:00.381310 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:00.381780 (+ 470us) server_negotiation.cc:299] Negotiation successful
0926 16:34:00.382040 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":423,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.60.keytab.
Entry for principal kudu/127.6.33.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.60.keytab.
I20260926 16:34:00.385115 19850 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.33.60" port: 37327 } has no permanent_uuid. Determining permanent_uuid...
I20260926 16:34:00.386267 19853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:00.385722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:37327 (local address 127.0.0.1:53868)
0926 16:34:00.385928 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:00.386070 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.60:37327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 16:34:00.386245 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.60:37327
--webserver_interface=127.6.33.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.60.keytab
--principal=kudu/127.6.33.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:42373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.33.62:34199,127.6.33.61:38883,127.6.33.60:37327
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:34:00.386551 19799 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.33.60:37327: connect: Connection refused (error 111)
I20260926 16:34:00.388396 19853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:00.387956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:37327 (local address 127.0.0.1:53882)
0926 16:34:00.388134 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:00.388241 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.60:37327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 16:34:00.388607 19850 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.33.60:37327: Network error: Client connection negotiation failed: client connection to 127.6.33.60:37327: connect: Connection refused (error 111)
I20260926 16:34:00.438014 19850 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.33.60" port: 37327 } attempt: 1
I20260926 16:34:00.439309 19853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:00.438834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:37327 (local address 127.0.0.1:53888)
0926 16:34:00.439039 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:00.439161 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.60:37327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260926 16:34:00.440290 19853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:00.439946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:37327 (local address 127.0.0.1:53902)
0926 16:34:00.440092 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:00.440187 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.60:37327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 16:34:00.440490 19850 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.33.60:37327: Network error: Client connection negotiation failed: client connection to 127.6.33.60:37327: connect: Connection refused (error 111)
I20260926 16:34:00.508725 19850 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.33.60" port: 37327 } attempt: 2
I20260926 16:34:00.509941 19853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:00.509466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:37327 (local address 127.0.0.1:53910)
0926 16:34:00.509693 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:00.509793 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.60:37327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260926 16:34:00.510951 19853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:00.510633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:37327 (local address 127.0.0.1:53916)
0926 16:34:00.510782 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:00.510845 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.60:37327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 16:34:00.511157 19850 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.33.60:37327: Network error: Client connection negotiation failed: client connection to 127.6.33.60:37327: connect: Connection refused (error 111)
I20260926 16:34:00.539433 19776 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.33.61" port: 38883 } attempt: 5
Sep 26 16:34:00 dist-test-slave-khd3 krb5kdc[19696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440439, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.62@KRBTEST.COM for kudu/127.6.33.61@KRBTEST.COM
I20260926 16:34:00.555219 19855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:00.540370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54612 (local address 127.6.33.61:38883)
0926 16:34:00.540645 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:00.540654 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:00.540682 (+ 28us) server_negotiation.cc:407] Connection header received
0926 16:34:00.540791 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:00.540798 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:00.540918 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:00.541063 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:34:00.541955 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:00.543259 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:00.544480 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:00.545223 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:00.545364 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:00.548847 (+ 3483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:34:00.548853 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:34:00.548858 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:34:00.548899 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:34:00.552308 (+ 3409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:00.553244 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:00.553250 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:00.553256 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:00.553387 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:00.553805 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:00.553810 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:00.553813 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:00.554111 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:34:00.554261 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:00.554662 (+ 401us) server_negotiation.cc:299] Negotiation successful
0926 16:34:00.554895 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 16:34:00.555267 19778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:00.540241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.61:38883 (local address 127.0.0.1:54612)
0926 16:34:00.540462 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:00.540486 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 16:34:00.540607 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:34:00.541147 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:34:00.541152 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:34:00.541580 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:34:00.541765 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:00.541775 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:00.543452 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:00.543456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:00.544319 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:00.544333 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:00.545417 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:00.545422 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:00.545590 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:00.546299 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:34:00.546317 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:34:00.548612 (+ 2295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:34:00.552484 (+ 3872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:34:00.552492 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:34:00.552495 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:34:00.553089 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:34:00.553546 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:34:00.553550 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:34:00.553552 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:34:00.553691 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:34:00.554241 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:34:00.554245 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:34:00.554402 (+ 157us) client_negotiation.cc:769] Sending connection context
0926 16:34:00.554561 (+ 159us) client_negotiation.cc:240] Negotiation successful
0926 16:34:00.554867 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260926 16:34:00.556573 19776 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.33.60" port: 37327 } has no permanent_uuid. Determining permanent_uuid...
I20260926 16:34:00.557622 19778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:00.557217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:37327 (local address 127.0.0.1:53922)
0926 16:34:00.557383 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:00.557477 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.60:37327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 16:34:00.558589 19778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:00.558272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:37327 (local address 127.0.0.1:53928)
0926 16:34:00.558421 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:00.558497 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.60:37327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 16:34:00.558779 19776 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.33.60:37327: Network error: Client connection negotiation failed: client connection to 127.6.33.60:37327: connect: Connection refused (error 111)
I20260926 16:34:00.602362 19850 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.33.60" port: 37327 } attempt: 3
I20260926 16:34:00.603827 19853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:00.603079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:37327 (local address 127.0.0.1:53930)
0926 16:34:00.603436 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:00.603598 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.60:37327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260926 16:34:00.605607 19853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:00.604759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:37327 (local address 127.0.0.1:53946)
0926 16:34:00.605159 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:00.605337 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.60:37327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165}
W20260926 16:34:00.605959 19850 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.33.60:37327: Network error: Client connection negotiation failed: client connection to 127.6.33.60:37327: connect: Connection refused (error 111)
I20260926 16:34:00.609997 19776 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.33.60" port: 37327 } attempt: 1
I20260926 16:34:00.611052 19778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:00.610614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:37327 (local address 127.0.0.1:53950)
0926 16:34:00.610796 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:00.610895 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.60:37327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 16:34:00.612015 19778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:00.611695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:37327 (local address 127.0.0.1:53956)
0926 16:34:00.611839 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:00.611914 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.60:37327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 16:34:00.612214 19776 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.33.60:37327: Network error: Client connection negotiation failed: client connection to 127.6.33.60:37327: connect: Connection refused (error 111)
I20260926 16:34:00.680452 19776 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.33.60" port: 37327 } attempt: 2
I20260926 16:34:00.681656 19778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:00.681161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:37327 (local address 127.0.0.1:53968)
0926 16:34:00.681386 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:00.681477 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.60:37327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260926 16:34:00.682662 19778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:00.682296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:37327 (local address 127.0.0.1:53978)
0926 16:34:00.682468 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:00.682549 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.60:37327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260926 16:34:00.682852 19776 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.33.60:37327: Network error: Client connection negotiation failed: client connection to 127.6.33.60:37327: connect: Connection refused (error 111)
W20260926 16:34:00.719658 19857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:34:00.719998 19857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:34:00.720172 19857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:34:00.727490 19857 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:34:00.727636 19857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:34:00.727746 19857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:34:00.727849 19857 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:34:00.727926 19857 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:34:00.738262 19857 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:42373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.33.62:34199,127.6.33.61:38883,127.6.33.60:37327
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.60.keytab
--principal=kudu/127.6.33.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.60:37327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.33.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:34:00.739679 19857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:34:00.741040 19857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 16:34:00.749212 19850 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.33.60" port: 37327 } attempt: 4
I20260926 16:34:00.750352 19853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:00.749898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:37327 (local address 127.0.0.1:53994)
0926 16:34:00.750119 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:00.750205 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.60:37327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260926 16:34:00.751477 19853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:00.751045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:37327 (local address 127.0.0.1:54000)
0926 16:34:00.751224 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:00.751325 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.60:37327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260926 16:34:00.751835 19850 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.33.60:37327: Network error: Client connection negotiation failed: client connection to 127.6.33.60:37327: connect: Connection refused (error 111)
W20260926 16:34:00.753057 19863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:34:00.753125 19865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:34:00.753147 19862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:34:00.754674 19857 server_base.cc:1034] running on GCE node
I20260926 16:34:00.755487 19857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:34:00.756979 19857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:34:00.758236 19857 hybrid_clock.cc:648] HybridClock initialized: now 1790440440758183 us; error 43 us; skew 500 ppm
I20260926 16:34:00.758620 19857 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:34:00 dist-test-slave-khd3 krb5kdc[19696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440440, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:34:00.761759 19857 init.cc:377] Logged in from keytab as kudu/127.6.33.60@KRBTEST.COM (short username kudu)
I20260926 16:34:00.763191 19857 webserver.cc:460] Webserver started at http://127.6.33.60:38257/ using document root <none> and password file <none>
I20260926 16:34:00.763895 19857 fs_manager.cc:359] Metadata directory not provided
I20260926 16:34:00.764038 19857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:34:00.764335 19857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:34:00.767379 19857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/minicluster-data/master-2/data/instance:
uuid: "671fdaa7ecc74df3ac97579314a689fd"
format_stamp: "Formatted at 2026-09-26 16:34:00 on dist-test-slave-khd3"
server_key: "3759ecc4bdefd187de49db436132d0b6"
server_key_iv: "3043f43df78623d98e3acc3f6ea905a5"
server_key_version: "encryptionkey@0"
I20260926 16:34:00.768191 19857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/minicluster-data/master-2/wal/instance:
uuid: "671fdaa7ecc74df3ac97579314a689fd"
format_stamp: "Formatted at 2026-09-26 16:34:00 on dist-test-slave-khd3"
server_key: "3759ecc4bdefd187de49db436132d0b6"
server_key_iv: "3043f43df78623d98e3acc3f6ea905a5"
server_key_version: "encryptionkey@0"
I20260926 16:34:00.773371 19857 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260926 16:34:00.777117 19872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:34:00.778570 19857 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:34:00.778776 19857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/minicluster-data/master-2/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/minicluster-data/master-2/wal
uuid: "671fdaa7ecc74df3ac97579314a689fd"
format_stamp: "Formatted at 2026-09-26 16:34:00 on dist-test-slave-khd3"
server_key: "3759ecc4bdefd187de49db436132d0b6"
server_key_iv: "3043f43df78623d98e3acc3f6ea905a5"
server_key_version: "encryptionkey@0"
I20260926 16:34:00.778996 19857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:34:00.789044 19776 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.33.60" port: 37327 } attempt: 3
I20260926 16:34:00.790159 19778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:00.789685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:37327 (local address 127.0.0.1:54010)
0926 16:34:00.789895 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:00.789990 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.60:37327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 16:34:00.791177 19778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:00.790798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:37327 (local address 127.0.0.1:54016)
0926 16:34:00.790989 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:00.791061 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.60:37327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260926 16:34:00.791383 19776 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.33.60:37327: Network error: Client connection negotiation failed: client connection to 127.6.33.60:37327: connect: Connection refused (error 111)
I20260926 16:34:00.806331 19857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:34:00.810252 19857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:34:00.810537 19857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:34:00.828841 19857 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.60:37327
I20260926 16:34:00.828886 19923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.60:37327 every 8 connection(s)
I20260926 16:34:00.830260 19857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/minicluster-data/master-2/data/info.pb
I20260926 16:34:00.834228 19924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:34:00.838672 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 19857
I20260926 16:34:00.838860 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440233850748-6276-0/minicluster-data/master-2/wal/instance
I20260926 16:34:00.839421 6276 external_mini_cluster.cc:1428] Setting key 1d73c6ee97c5fbadf463f1694b18fa9c
I20260926 16:34:00.841831 19924 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.33.62" port: 34199 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 16:34:00 dist-test-slave-khd3 krb5kdc[19696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440440, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.60@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
Sep 26 16:34:00 dist-test-slave-khd3 krb5kdc[19696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.60@KRBTEST.COM
I20260926 16:34:00.862427 19779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:00.846124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58444 (local address 127.6.33.62:34199)
0926 16:34:00.846362 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:00.846371 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:00.846944 (+ 573us) server_negotiation.cc:407] Connection header received
0926 16:34:00.847161 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:00.847166 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:00.847237 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:00.847363 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:34:00.848620 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:00.849342 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:00.850972 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:00.851565 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:00.851660 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:00.855770 (+ 4110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:34:00.855774 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:34:00.855777 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:34:00.855808 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:34:00.859174 (+ 3366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:00.860112 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:00.860121 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:00.860127 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:00.860280 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:00.860679 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:00.860684 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:00.860686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:00.861022 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:34:00.861186 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:00.861846 (+ 660us) server_negotiation.cc:299] Negotiation successful
0926 16:34:00.862067 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 16:34:00.863510 19928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:00.846084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:34199 (local address 127.0.0.1:58444)
0926 16:34:00.846807 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:00.846847 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 16:34:00.847031 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:34:00.847421 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:34:00.847427 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:34:00.848082 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:34:00.848469 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:00.848482 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:00.849508 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:00.849513 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:00.850791 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:00.850807 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:00.851813 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:00.851821 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:00.852034 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:00.852681 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:34:00.852718 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:34:00.855578 (+ 2860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:34:00.859321 (+ 3743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:34:00.859330 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:34:00.859340 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:34:00.859992 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:34:00.860391 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:34:00.860396 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:34:00.860399 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:34:00.860570 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:34:00.861149 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:34:00.861156 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:34:00.861503 (+ 347us) client_negotiation.cc:769] Sending connection context
0926 16:34:00.861759 (+ 256us) client_negotiation.cc:240] Negotiation successful
0926 16:34:00.862247 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":387,"threads_started":1}
I20260926 16:34:00.863590 19927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:00.841385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54030 (local address 127.6.33.60:37327)
0926 16:34:00.842200 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:00.842212 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:00.842254 (+ 42us) server_negotiation.cc:407] Connection header received
0926 16:34:00.843839 (+ 1585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:00.843863 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:00.844331 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:00.844663 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:34:00.846504 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:00.848009 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:00.849632 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:00.850283 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:00.850405 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:00.856377 (+ 5972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:34:00.856384 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:34:00.856394 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:34:00.856445 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:34:00.860028 (+ 3583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:00.860833 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:00.860840 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:00.860842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:00.860987 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:00.861399 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:00.861409 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:00.861411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:00.861745 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:34:00.862079 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:00.862456 (+ 377us) server_negotiation.cc:299] Negotiation successful
0926 16:34:00.862729 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":393,"threads_started":1}
I20260926 16:34:00.865242 19924 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.33.61" port: 38883 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 16:34:00 dist-test-slave-khd3 krb5kdc[19696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440440, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.60@KRBTEST.COM for kudu/127.6.33.61@KRBTEST.COM
Sep 26 16:34:00 dist-test-slave-khd3 krb5kdc[19696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:34:00.880457 19855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:00.866165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54628 (local address 127.6.33.61:38883)
0926 16:34:00.866349 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:00.866354 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:00.866433 (+ 79us) server_negotiation.cc:407] Connection header received
0926 16:34:00.866643 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:00.866648 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:00.866732 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:00.866804 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:34:00.867754 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:00.868739 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:00.870263 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:00.870930 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:00.871066 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:00.874491 (+ 3425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:34:00.874496 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:34:00.874500 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:34:00.874535 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:34:00.877659 (+ 3124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:00.878396 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:00.878404 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:00.878408 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:00.878572 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:00.878984 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:00.878991 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:00.878995 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:00.879275 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:34:00.879447 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:00.879908 (+ 461us) server_negotiation.cc:299] Negotiation successful
0926 16:34:00.880125 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 16:34:00.880509 19928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:00.866155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.61:38883 (local address 127.0.0.1:54628)
0926 16:34:00.866343 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:00.866364 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 16:34:00.866479 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:34:00.867053 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:34:00.867058 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:34:00.867456 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:34:00.867594 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:00.867602 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:00.868928 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:00.868934 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:00.870127 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:00.870138 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:00.871106 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:00.871110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:00.871300 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:00.871920 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:34:00.871944 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:34:00.874293 (+ 2349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:34:00.877810 (+ 3517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:34:00.877815 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:34:00.877818 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:34:00.878245 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:34:00.878700 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:34:00.878705 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:34:00.878710 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:34:00.878860 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:34:00.879403 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:34:00.879409 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:34:00.879637 (+ 228us) client_negotiation.cc:769] Sending connection context
0926 16:34:00.879817 (+ 180us) client_negotiation.cc:240] Negotiation successful
0926 16:34:00.880127 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260926 16:34:00.881862 19924 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.33.60" port: 37327 } has no permanent_uuid. Determining permanent_uuid...
I20260926 16:34:00.882649 19779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:00.866630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58450 (local address 127.6.33.62:34199)
0926 16:34:00.866817 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:00.866822 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:00.866839 (+ 17us) server_negotiation.cc:407] Connection header received
0926 16:34:00.866975 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:00.866979 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:00.867052 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:00.867145 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:34:00.868511 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:00.869619 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:00.871323 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:00.871916 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:00.872047 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:00.876677 (+ 4630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:34:00.876684 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:34:00.876689 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:34:00.876732 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:34:00.879482 (+ 2750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:00.880347 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:00.880352 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:00.880355 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:00.880509 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:00.881017 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:00.881026 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:00.881031 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:00.881395 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:34:00.881717 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:00.882064 (+ 347us) server_negotiation.cc:299] Negotiation successful
0926 16:34:00.882296 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
Sep 26 16:34:00 dist-test-slave-khd3 krb5kdc[19696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440440, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.60@KRBTEST.COM for kudu/127.6.33.60@KRBTEST.COM
Sep 26 16:34:00 dist-test-slave-khd3 krb5kdc[19696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.61@KRBTEST.COM
I20260926 16:34:00.897136 19928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:00.882734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:37327 (local address 127.0.0.1:54032)
0926 16:34:00.882992 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:00.883010 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 16:34:00.883125 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:34:00.883416 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:34:00.883420 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:34:00.883928 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:34:00.884150 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:00.884160 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:00.885647 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:00.885650 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:00.886564 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:00.886582 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:00.887403 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:00.887410 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:00.887613 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:00.888221 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:34:00.888309 (+ 88us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:34:00.890534 (+ 2225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:34:00.894518 (+ 3984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:34:00.894526 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:34:00.894530 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:34:00.894897 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:34:00.895296 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:34:00.895303 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:34:00.895306 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:34:00.895443 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:34:00.895963 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:34:00.895968 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:34:00.896124 (+ 156us) client_negotiation.cc:769] Sending connection context
0926 16:34:00.896282 (+ 158us) client_negotiation.cc:240] Negotiation successful
0926 16:34:00.896621 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139}
I20260926 16:34:00.897135 19927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:00.882802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54032 (local address 127.6.33.60:37327)
0926 16:34:00.883046 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:00.883053 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:00.883071 (+ 18us) server_negotiation.cc:407] Connection header received
0926 16:34:00.883242 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:00.883248 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:00.883313 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:00.883404 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:34:00.884320 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:00.885518 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:00.886719 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:00.887267 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:00.887365 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:00.890748 (+ 3383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:34:00.890754 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:34:00.890759 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:34:00.890804 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:34:00.894384 (+ 3580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:00.895032 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:00.895038 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:00.895041 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:00.895175 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:00.895560 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:00.895566 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:00.895569 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:00.895844 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:34:00.896003 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:00.896368 (+ 365us) server_negotiation.cc:299] Negotiation successful
0926 16:34:00.896804 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":165,"server-negotiator.queue_time_us":154}
I20260926 16:34:00.900725 19855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:00.884579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54640 (local address 127.6.33.61:38883)
0926 16:34:00.884749 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:00.884790 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:00.884812 (+ 22us) server_negotiation.cc:407] Connection header received
0926 16:34:00.884881 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:00.884884 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:00.884936 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:00.885001 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:34:00.886459 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:00.887377 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:00.889236 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:00.889724 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:00.889818 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:00.895316 (+ 5498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:34:00.895321 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:34:00.895324 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:34:00.895354 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:34:00.898140 (+ 2786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:00.898811 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:00.898816 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:00.898818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:00.898946 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:00.899343 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:00.899347 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:00.899349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:00.899595 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:34:00.899872 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:00.900209 (+ 337us) server_negotiation.cc:299] Negotiation successful
0926 16:34:00.900429 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 16:34:00.902379 19924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 671fdaa7ecc74df3ac97579314a689fd: Bootstrap starting.
I20260926 16:34:00.905618 6276 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:34:00.906347 19924 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 671fdaa7ecc74df3ac97579314a689fd: Neither blocks nor log segments found. Creating new log.
I20260926 16:34:00.907410 19924 log.cc:824] T 00000000000000000000000000000000 P 671fdaa7ecc74df3ac97579314a689fd: Log is configured to *not* fsync() on all Append() calls
I20260926 16:34:00.910830 19924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 671fdaa7ecc74df3ac97579314a689fd: No bootstrap required, opened a new log
I20260926 16:34:00.921221 19924 raft_consensus.cc:348] T 00000000000000000000000000000000 P 671fdaa7ecc74df3ac97579314a689fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32d712db95ac4b0e9ed8b8574ef883d5" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 34199 } } peers { permanent_uuid: "bf27e4b5f3c9462fa5e10619b37c3a58" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 38883 } } peers { permanent_uuid: "671fdaa7ecc74df3ac97579314a689fd" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 37327 } }
I20260926 16:34:00.921697 19924 raft_consensus.cc:374] T 00000000000000000000000000000000 P 671fdaa7ecc74df3ac97579314a689fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:34:00.921823 19924 raft_consensus.cc:729] T 00000000000000000000000000000000 P 671fdaa7ecc74df3ac97579314a689fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 671fdaa7ecc74df3ac97579314a689fd, State: Initialized, Role: FOLLOWER
I20260926 16:34:00.922669 19924 consensus_queue.cc:260] T 00000000000000000000000000000000 P 671fdaa7ecc74df3ac97579314a689fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32d712db95ac4b0e9ed8b8574ef883d5" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 34199 } } peers { permanent_uuid: "bf27e4b5f3c9462fa5e10619b37c3a58" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 38883 } } peers { permanent_uuid: "671fdaa7ecc74df3ac97579314a689fd" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 37327 } }
I20260926 16:34:00.923996 19937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 671fdaa7ecc74df3ac97579314a689fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32d712db95ac4b0e9ed8b8574ef883d5" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 34199 } } peers { permanent_uuid: "bf27e4b5f3c9462fa5e10619b37c3a58" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 38883 } } peers { permanent_uuid: "671fdaa7ecc74df3ac97579314a689fd" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 37327 } } }
I20260926 16:34:00.924583 19937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 671fdaa7ecc74df3ac97579314a689fd [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:34:00.925473 19924 sys_catalog.cc:564] T 00000000000000000000000000000000 P 671fdaa7ecc74df3ac97579314a689fd [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:34:00.935025 19779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:00.910860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58456 (local address 127.6.33.62:34199)
0926 16:34:00.911161 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:00.911171 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:00.913389 (+ 2218us) server_negotiation.cc:407] Connection header received
0926 16:34:00.913674 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:00.913679 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:00.913766 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:00.913835 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:34:00.916028 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:00.917184 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:00.919241 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:00.919881 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:00.919978 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:00.924104 (+ 4126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:34:00.924111 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:34:00.924115 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:34:00.924154 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:34:00.929726 (+ 5572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:00.931205 (+ 1479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:00.931214 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:00.931220 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:00.931414 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:00.932011 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:00.932019 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:00.932025 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:00.932482 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:34:00.934034 (+ 1552us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:00.934215 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 16:34:00.934523 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260926 16:34:00.936072 19855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:00.910674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54648 (local address 127.6.33.61:38883)
0926 16:34:00.911071 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:00.911077 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:00.913803 (+ 2726us) server_negotiation.cc:407] Connection header received
0926 16:34:00.914080 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:00.914086 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:00.914168 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:00.914250 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:34:00.917183 (+ 2933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:00.918757 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:00.921610 (+ 2853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:00.922380 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:00.922502 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:00.927482 (+ 4980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:34:00.927489 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:34:00.927496 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:34:00.927542 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:34:00.931678 (+ 4136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:00.932841 (+ 1163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:00.932847 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:00.932850 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:00.933016 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:00.933494 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:00.933502 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:00.933507 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:00.933961 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:34:00.935213 (+ 1252us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:00.935375 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 16:34:00.935643 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258}
I20260926 16:34:00.940568 19927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:00.913891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54048 (local address 127.6.33.60:37327)
0926 16:34:00.914117 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:00.914123 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:00.921271 (+ 7148us) server_negotiation.cc:407] Connection header received
0926 16:34:00.921560 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:00.921570 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:00.921657 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:00.921736 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:34:00.924373 (+ 2637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:00.925670 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:00.927933 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:00.928977 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:00.929160 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:00.933080 (+ 3920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:34:00.933087 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:34:00.933096 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:34:00.933141 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:34:00.937504 (+ 4363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:00.938429 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:00.938439 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:00.938442 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:00.938575 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:00.938998 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:00.939004 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:00.939007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:00.939310 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:34:00.939584 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:00.939901 (+ 317us) server_negotiation.cc:299] Negotiation successful
0926 16:34:00.940189 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260926 16:34:00.948359 19948 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 671fdaa7ecc74df3ac97579314a689fd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 16:34:00.948586 19948 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 16:34:00.945478 19857 master_runner.cc:186] Error getting master registration for 127.6.33.62:34199: OK, instance_id { permanent_uuid: "32d712db95ac4b0e9ed8b8574ef883d5" instance_seqno: 1790440439825979 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 16:34:00.955983 19857 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 16:34:00.968602 19776 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.33.60" port: 37327 } attempt: 4
Sep 26 16:34:00 dist-test-slave-khd3 krb5kdc[19696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440439, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.62@KRBTEST.COM for kudu/127.6.33.60@KRBTEST.COM
I20260926 16:34:00.983871 19778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:00.969261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:37327 (local address 127.0.0.1:54062)
0926 16:34:00.969443 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:00.969470 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 16:34:00.969577 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:34:00.969979 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:34:00.969986 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:34:00.970357 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:34:00.970511 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:00.970518 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:00.972564 (+ 2046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:00.972570 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:00.973589 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:00.973601 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:00.974519 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:00.974523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:00.974653 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:00.975299 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:34:00.975316 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:34:00.977781 (+ 2465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:34:00.980974 (+ 3193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:34:00.980979 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:34:00.980984 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:34:00.981510 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:34:00.981993 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:34:00.981998 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:34:00.982002 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:34:00.982206 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:34:00.982866 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:34:00.982872 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:34:00.983033 (+ 161us) client_negotiation.cc:769] Sending connection context
0926 16:34:00.983170 (+ 137us) client_negotiation.cc:240] Negotiation successful
0926 16:34:00.983485 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 16:34:00.984453 19927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:00.969317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54062 (local address 127.6.33.60:37327)
0926 16:34:00.969502 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:00.969507 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:00.969527 (+ 20us) server_negotiation.cc:407] Connection header received
0926 16:34:00.969697 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:00.969704 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:00.969798 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:00.969896 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:34:00.970671 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:00.972414 (+ 1743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:00.973720 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:00.974388 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:00.974514 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:00.977962 (+ 3448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:34:00.977967 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:34:00.977970 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:34:00.977999 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:34:00.980835 (+ 2836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:00.981656 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:00.981662 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:00.981666 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:00.981846 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:00.982329 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:00.982336 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:00.982339 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:00.982713 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:34:00.983529 (+ 816us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:00.983671 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 16:34:00.984169 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 16:34:00.989274 19776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32d712db95ac4b0e9ed8b8574ef883d5: Bootstrap starting.
I20260926 16:34:00.992538 19776 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32d712db95ac4b0e9ed8b8574ef883d5: Neither blocks nor log segments found. Creating new log.
I20260926 16:34:00.994035 19776 log.cc:824] T 00000000000000000000000000000000 P 32d712db95ac4b0e9ed8b8574ef883d5: Log is configured to *not* fsync() on all Append() calls
I20260926 16:34:00.996913 19776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32d712db95ac4b0e9ed8b8574ef883d5: No bootstrap required, opened a new log
I20260926 16:34:01.004987 19776 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32d712db95ac4b0e9ed8b8574ef883d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32d712db95ac4b0e9ed8b8574ef883d5" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 34199 } } peers { permanent_uuid: "bf27e4b5f3c9462fa5e10619b37c3a58" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 38883 } } peers { permanent_uuid: "671fdaa7ecc74df3ac97579314a689fd" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 37327 } }
I20260926 16:34:01.005337 19776 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32d712db95ac4b0e9ed8b8574ef883d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:34:01.005482 19776 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32d712db95ac4b0e9ed8b8574ef883d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32d712db95ac4b0e9ed8b8574ef883d5, State: Initialized, Role: FOLLOWER
I20260926 16:34:01.006076 19776 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32d712db95ac4b0e9ed8b8574ef883d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32d712db95ac4b0e9ed8b8574ef883d5" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 34199 } } peers { permanent_uuid: "bf27e4b5f3c9462fa5e10619b37c3a58" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 38883 } } peers { permanent_uuid: "671fdaa7ecc74df3ac97579314a689fd" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 37327 } }
I20260926 16:34:01.007272 19951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32d712db95ac4b0e9ed8b8574ef883d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32d712db95ac4b0e9ed8b8574ef883d5" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 34199 } } peers { permanent_uuid: "bf27e4b5f3c9462fa5e10619b37c3a58" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 38883 } } peers { permanent_uuid: "671fdaa7ecc74df3ac97579314a689fd" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 37327 } } }
I20260926 16:34:01.007675 19951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32d712db95ac4b0e9ed8b8574ef883d5 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:34:01.008626 19776 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32d712db95ac4b0e9ed8b8574ef883d5 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 16:34:01.020907 19962 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 32d712db95ac4b0e9ed8b8574ef883d5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 16:34:01.021178 19962 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 16:34:01.019052 19709 master_runner.cc:186] Error getting master registration for 127.6.33.61:38883: OK, instance_id { permanent_uuid: "bf27e4b5f3c9462fa5e10619b37c3a58" instance_seqno: 1790440440316788 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 16:34:01.030503 19709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 16:34:01.051086 19850 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.33.60" port: 37327 } attempt: 5
Sep 26 16:34:01 dist-test-slave-khd3 krb5kdc[19696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440440, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.61@KRBTEST.COM for kudu/127.6.33.60@KRBTEST.COM
W20260926 16:34:01.062613 19878 tablet.cc:2369] T 00000000000000000000000000000000 P 671fdaa7ecc74df3ac97579314a689fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 16:34:01.065436 19730 tablet.cc:2369] T 00000000000000000000000000000000 P 32d712db95ac4b0e9ed8b8574ef883d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:34:01.069142 19853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:01.051868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:37327 (local address 127.0.0.1:54078)
0926 16:34:01.052170 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:01.052190 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 16:34:01.052324 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:34:01.052743 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:34:01.052748 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:34:01.053338 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:34:01.053556 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:01.053570 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:01.055164 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:01.055171 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:01.056684 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:01.056700 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:01.057981 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:01.057987 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:01.058186 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:01.058914 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:34:01.058939 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:34:01.061746 (+ 2807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:34:01.066308 (+ 4562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:34:01.066314 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:34:01.066318 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:34:01.066852 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:34:01.067270 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:34:01.067275 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:34:01.067279 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:34:01.067455 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:34:01.068075 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:34:01.068080 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:34:01.068248 (+ 168us) client_negotiation.cc:769] Sending connection context
0926 16:34:01.068403 (+ 155us) client_negotiation.cc:240] Negotiation successful
0926 16:34:01.068714 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176}
I20260926 16:34:01.069526 19927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:01.051971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54078 (local address 127.6.33.60:37327)
0926 16:34:01.052171 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:01.052177 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:01.052277 (+ 100us) server_negotiation.cc:407] Connection header received
0926 16:34:01.052472 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:01.052478 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:01.052562 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:01.052687 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:34:01.053740 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:01.054961 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:01.056927 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:01.057619 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:01.057751 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:01.061954 (+ 4203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:34:01.061963 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:34:01.061972 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:34:01.062013 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:34:01.066145 (+ 4132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:01.066996 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:01.067002 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:01.067006 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:01.067152 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:01.067585 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:01.067591 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:01.067594 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:01.067947 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:34:01.068112 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:01.068549 (+ 437us) server_negotiation.cc:299] Negotiation successful
0926 16:34:01.068843 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 16:34:01.075380 19850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf27e4b5f3c9462fa5e10619b37c3a58: Bootstrap starting.
I20260926 16:34:01.079542 19850 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf27e4b5f3c9462fa5e10619b37c3a58: Neither blocks nor log segments found. Creating new log.
I20260926 16:34:01.080900 19850 log.cc:824] T 00000000000000000000000000000000 P bf27e4b5f3c9462fa5e10619b37c3a58: Log is configured to *not* fsync() on all Append() calls
I20260926 16:34:01.084285 19850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf27e4b5f3c9462fa5e10619b37c3a58: No bootstrap required, opened a new log
I20260926 16:34:01.095716 19850 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf27e4b5f3c9462fa5e10619b37c3a58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32d712db95ac4b0e9ed8b8574ef883d5" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 34199 } } peers { permanent_uuid: "bf27e4b5f3c9462fa5e10619b37c3a58" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 38883 } } peers { permanent_uuid: "671fdaa7ecc74df3ac97579314a689fd" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 37327 } }
I20260926 16:34:01.096141 19850 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf27e4b5f3c9462fa5e10619b37c3a58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:34:01.096268 19850 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf27e4b5f3c9462fa5e10619b37c3a58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf27e4b5f3c9462fa5e10619b37c3a58, State: Initialized, Role: FOLLOWER
I20260926 16:34:01.096982 19850 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf27e4b5f3c9462fa5e10619b37c3a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32d712db95ac4b0e9ed8b8574ef883d5" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 34199 } } peers { permanent_uuid: "bf27e4b5f3c9462fa5e10619b37c3a58" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 38883 } } peers { permanent_uuid: "671fdaa7ecc74df3ac97579314a689fd" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 37327 } }
I20260926 16:34:01.098065 19965 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf27e4b5f3c9462fa5e10619b37c3a58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32d712db95ac4b0e9ed8b8574ef883d5" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 34199 } } peers { permanent_uuid: "bf27e4b5f3c9462fa5e10619b37c3a58" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 38883 } } peers { permanent_uuid: "671fdaa7ecc74df3ac97579314a689fd" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 37327 } } }
I20260926 16:34:01.098541 19965 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf27e4b5f3c9462fa5e10619b37c3a58 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:34:01.099413 19850 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf27e4b5f3c9462fa5e10619b37c3a58 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:34:01.117380 19783 master_runner.cc:263] No leader master found from master bf27e4b5f3c9462fa5e10619b37c3a58
I20260926 16:34:01.118079 19783 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 16:34:01.120214 19976 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bf27e4b5f3c9462fa5e10619b37c3a58: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 16:34:01.120447 19976 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 16:34:01.145133 19965 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf27e4b5f3c9462fa5e10619b37c3a58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:34:01.145447 19965 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf27e4b5f3c9462fa5e10619b37c3a58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32d712db95ac4b0e9ed8b8574ef883d5" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 34199 } } peers { permanent_uuid: "bf27e4b5f3c9462fa5e10619b37c3a58" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 38883 } } peers { permanent_uuid: "671fdaa7ecc74df3ac97579314a689fd" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 37327 } }
I20260926 16:34:01.147079 19965 leader_election.cc:288] T 00000000000000000000000000000000 P bf27e4b5f3c9462fa5e10619b37c3a58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 32d712db95ac4b0e9ed8b8574ef883d5 (127.6.33.62:34199), 671fdaa7ecc74df3ac97579314a689fd (127.6.33.60:37327)
I20260926 16:34:01.148231 19751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bf27e4b5f3c9462fa5e10619b37c3a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32d712db95ac4b0e9ed8b8574ef883d5" is_pre_election: true
I20260926 16:34:01.148615 19899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bf27e4b5f3c9462fa5e10619b37c3a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "671fdaa7ecc74df3ac97579314a689fd" is_pre_election: true
I20260926 16:34:01.148921 19751 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 32d712db95ac4b0e9ed8b8574ef883d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf27e4b5f3c9462fa5e10619b37c3a58 in term 0.
I20260926 16:34:01.149235 19899 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 671fdaa7ecc74df3ac97579314a689fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf27e4b5f3c9462fa5e10619b37c3a58 in term 0.
I20260926 16:34:01.149593 19801 leader_election.cc:302] T 00000000000000000000000000000000 P bf27e4b5f3c9462fa5e10619b37c3a58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32d712db95ac4b0e9ed8b8574ef883d5, bf27e4b5f3c9462fa5e10619b37c3a58; no voters:
I20260926 16:34:01.150002 19965 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf27e4b5f3c9462fa5e10619b37c3a58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:34:01.150146 19965 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf27e4b5f3c9462fa5e10619b37c3a58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:34:01.150238 19965 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf27e4b5f3c9462fa5e10619b37c3a58 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:34:01.154160 19965 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf27e4b5f3c9462fa5e10619b37c3a58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32d712db95ac4b0e9ed8b8574ef883d5" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 34199 } } peers { permanent_uuid: "bf27e4b5f3c9462fa5e10619b37c3a58" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 38883 } } peers { permanent_uuid: "671fdaa7ecc74df3ac97579314a689fd" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 37327 } }
I20260926 16:34:01.155303 19751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bf27e4b5f3c9462fa5e10619b37c3a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32d712db95ac4b0e9ed8b8574ef883d5"
I20260926 16:34:01.155561 19751 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32d712db95ac4b0e9ed8b8574ef883d5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:34:01.159129 19751 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 32d712db95ac4b0e9ed8b8574ef883d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf27e4b5f3c9462fa5e10619b37c3a58 in term 1.
I20260926 16:34:01.161296 19801 leader_election.cc:302] T 00000000000000000000000000000000 P bf27e4b5f3c9462fa5e10619b37c3a58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32d712db95ac4b0e9ed8b8574ef883d5, bf27e4b5f3c9462fa5e10619b37c3a58; no voters:
I20260926 16:34:01.164673 19899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bf27e4b5f3c9462fa5e10619b37c3a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "671fdaa7ecc74df3ac97579314a689fd"
I20260926 16:34:01.164939 19899 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 671fdaa7ecc74df3ac97579314a689fd [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:34:01.168363 19899 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 671fdaa7ecc74df3ac97579314a689fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf27e4b5f3c9462fa5e10619b37c3a58 in term 1.
I20260926 16:34:01.169112 19965 leader_election.cc:288] T 00000000000000000000000000000000 P bf27e4b5f3c9462fa5e10619b37c3a58 [CANDIDATE]: Term 1 election: Requested vote from peers 32d712db95ac4b0e9ed8b8574ef883d5 (127.6.33.62:34199), 671fdaa7ecc74df3ac97579314a689fd (127.6.33.60:37327)
I20260926 16:34:01.169338 19965 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf27e4b5f3c9462fa5e10619b37c3a58 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:34:01.169483 19965 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf27e4b5f3c9462fa5e10619b37c3a58 [term 1 LEADER]: Becoming Leader. State: Replica: bf27e4b5f3c9462fa5e10619b37c3a58, State: Running, Role: LEADER
I20260926 16:34:01.170082 19965 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf27e4b5f3c9462fa5e10619b37c3a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32d712db95ac4b0e9ed8b8574ef883d5" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 34199 } } peers { permanent_uuid: "bf27e4b5f3c9462fa5e10619b37c3a58" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 38883 } } peers { permanent_uuid: "671fdaa7ecc74df3ac97579314a689fd" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 37327 } }
I20260926 16:34:01.192301 19978 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf27e4b5f3c9462fa5e10619b37c3a58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf27e4b5f3c9462fa5e10619b37c3a58. Latest consensus state: current_term: 1 leader_uuid: "bf27e4b5f3c9462fa5e10619b37c3a58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32d712db95ac4b0e9ed8b8574ef883d5" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 34199 } } peers { permanent_uuid: "bf27e4b5f3c9462fa5e10619b37c3a58" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 38883 } } peers { permanent_uuid: "671fdaa7ecc74df3ac97579314a689fd" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 37327 } } }
I20260926 16:34:01.192775 19978 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf27e4b5f3c9462fa5e10619b37c3a58 [sys.catalog]: This master's current role is: LEADER
I20260926 16:34:01.205437 19980 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:34:01.207350 19980 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:34:01.226472 19899 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 671fdaa7ecc74df3ac97579314a689fd [term 1 FOLLOWER]: Refusing update from remote peer bf27e4b5f3c9462fa5e10619b37c3a58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:34:01.227466 19978 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bf27e4b5f3c9462fa5e10619b37c3a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "671fdaa7ecc74df3ac97579314a689fd" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 37327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:34:01.234876 19751 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 32d712db95ac4b0e9ed8b8574ef883d5 [term 1 FOLLOWER]: Refusing update from remote peer bf27e4b5f3c9462fa5e10619b37c3a58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:34:01.235728 19965 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bf27e4b5f3c9462fa5e10619b37c3a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32d712db95ac4b0e9ed8b8574ef883d5" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 34199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:34:01.258280 19937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 671fdaa7ecc74df3ac97579314a689fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf27e4b5f3c9462fa5e10619b37c3a58. Latest consensus state: current_term: 1 leader_uuid: "bf27e4b5f3c9462fa5e10619b37c3a58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32d712db95ac4b0e9ed8b8574ef883d5" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 34199 } } peers { permanent_uuid: "bf27e4b5f3c9462fa5e10619b37c3a58" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 38883 } } peers { permanent_uuid: "671fdaa7ecc74df3ac97579314a689fd" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 37327 } } }
I20260926 16:34:01.258648 19937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 671fdaa7ecc74df3ac97579314a689fd [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:34:01.266404 19981 mvcc.cc:205] Tried to move back new op lower bound from 7333644047250984960 to 7333644047046434816. Current Snapshot: MvccSnapshot[applied={T|T < 7333644047250984960}]
I20260926 16:34:01.282866 19937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 671fdaa7ecc74df3ac97579314a689fd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bf27e4b5f3c9462fa5e10619b37c3a58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32d712db95ac4b0e9ed8b8574ef883d5" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 34199 } } peers { permanent_uuid: "bf27e4b5f3c9462fa5e10619b37c3a58" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 38883 } } peers { permanent_uuid: "671fdaa7ecc74df3ac97579314a689fd" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 37327 } } }
I20260926 16:34:01.283210 19937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 671fdaa7ecc74df3ac97579314a689fd [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:34:01.286573 19951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32d712db95ac4b0e9ed8b8574ef883d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf27e4b5f3c9462fa5e10619b37c3a58. Latest consensus state: current_term: 1 leader_uuid: "bf27e4b5f3c9462fa5e10619b37c3a58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32d712db95ac4b0e9ed8b8574ef883d5" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 34199 } } peers { permanent_uuid: "bf27e4b5f3c9462fa5e10619b37c3a58" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 38883 } } peers { permanent_uuid: "671fdaa7ecc74df3ac97579314a689fd" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 37327 } } }
I20260926 16:34:01.286931 19951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32d712db95ac4b0e9ed8b8574ef883d5 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:34:01.288226 19965 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf27e4b5f3c9462fa5e10619b37c3a58 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bf27e4b5f3c9462fa5e10619b37c3a58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32d712db95ac4b0e9ed8b8574ef883d5" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 34199 } } peers { permanent_uuid: "bf27e4b5f3c9462fa5e10619b37c3a58" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 38883 } } peers { permanent_uuid: "671fdaa7ecc74df3ac97579314a689fd" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 37327 } } }
I20260926 16:34:01.288575 19965 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf27e4b5f3c9462fa5e10619b37c3a58 [sys.catalog]: This master's current role is: LEADER
I20260926 16:34:01.293000 19980 catalog_manager.cc:1347] Generated new cluster ID: 64462467efaf490bbc9e94a565eefd45
I20260926 16:34:01.293107 19980 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:34:01.293562 19983 mvcc.cc:205] Tried to move back new op lower bound from 7333644047250984960 to 7333644047046434816. Current Snapshot: MvccSnapshot[applied={T|T < 7333644047250984960}]
I20260926 16:34:01.297112 19978 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf27e4b5f3c9462fa5e10619b37c3a58 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bf27e4b5f3c9462fa5e10619b37c3a58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32d712db95ac4b0e9ed8b8574ef883d5" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 34199 } } peers { permanent_uuid: "bf27e4b5f3c9462fa5e10619b37c3a58" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 38883 } } peers { permanent_uuid: "671fdaa7ecc74df3ac97579314a689fd" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 37327 } } }
I20260926 16:34:01.297389 19978 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf27e4b5f3c9462fa5e10619b37c3a58 [sys.catalog]: This master's current role is: LEADER
I20260926 16:34:01.305948 19951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32d712db95ac4b0e9ed8b8574ef883d5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bf27e4b5f3c9462fa5e10619b37c3a58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32d712db95ac4b0e9ed8b8574ef883d5" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 34199 } } peers { permanent_uuid: "bf27e4b5f3c9462fa5e10619b37c3a58" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 38883 } } peers { permanent_uuid: "671fdaa7ecc74df3ac97579314a689fd" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 37327 } } }
I20260926 16:34:01.306221 19951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32d712db95ac4b0e9ed8b8574ef883d5 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:34:01.320001 19980 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:34:01.322073 19980 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:34:01.339308 19980 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf27e4b5f3c9462fa5e10619b37c3a58: Generated new TSK 0
I20260926 16:34:01.340454 19980 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:34:01.950490 19948 catalog_manager.cc:1259] Loaded cluster ID: 64462467efaf490bbc9e94a565eefd45
I20260926 16:34:01.950583 19948 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 671fdaa7ecc74df3ac97579314a689fd: loading cluster ID for follower catalog manager: success
I20260926 16:34:01.952948 19948 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 671fdaa7ecc74df3ac97579314a689fd: acquiring CA information for follower catalog manager: success
I20260926 16:34:01.954674 19948 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 671fdaa7ecc74df3ac97579314a689fd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 16:34:02.023226 19962 catalog_manager.cc:1259] Loaded cluster ID: 64462467efaf490bbc9e94a565eefd45
I20260926 16:34:02.023380 19962 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 32d712db95ac4b0e9ed8b8574ef883d5: loading cluster ID for follower catalog manager: success
I20260926 16:34:02.025952 19962 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 32d712db95ac4b0e9ed8b8574ef883d5: acquiring CA information for follower catalog manager: success
I20260926 16:34:02.027868 19962 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 32d712db95ac4b0e9ed8b8574ef883d5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 16:34:02.463272 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 19709
I20260926 16:34:02.464579 19771 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:34:02.579571 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 19709
I20260926 16:34:02.600919 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 19783
I20260926 16:34:02.602315 19845 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:34:02.712491 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 19783
I20260926 16:34:02.733060 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 19857
I20260926 16:34:02.734284 19919 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:34:02.843640 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 19857
2026-09-26T16:34:02Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4801 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440233850748-6276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:34:02 dist-test-slave-khd3 krb5kdc[19998](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:34:02 dist-test-slave-khd3 krb5kdc[19998](info): set up 2 sockets
Sep 26 16:34:02 dist-test-slave-khd3 krb5kdc[19998](info): commencing operation
krb5kdc: starting...
W20260926 16:34:03.966040 6276 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Sep 26 16:34:04 dist-test-slave-khd3 krb5kdc[19998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:34:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:34:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab.
Entry for principal kudu/127.6.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab.
I20260926 16:34:04.206265 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440233850748-6276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440233850748-6276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440233850748-6276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:43961
--webserver_interface=127.6.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:40281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.33.62:43961
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:34:04.539846 20011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:34:04.540132 20011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:34:04.540220 20011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:34:04.547667 20011 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:34:04.547780 20011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:34:04.547827 20011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:34:04.547883 20011 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:34:04.547926 20011 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:34:04.558138 20011 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:40281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440233850748-6276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440233850748-6276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.33.62:43961
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:43961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440233850748-6276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:34:04.559442 20011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:34:04.560714 20011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:34:04.572818 20019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:34:04.572872 20017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:34:04.572844 20016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:34:04.574734 20011 server_base.cc:1034] running on GCE node
I20260926 16:34:04.575652 20011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:34:04.577255 20011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:34:04.578497 20011 hybrid_clock.cc:648] HybridClock initialized: now 1790440444578453 us; error 38 us; skew 500 ppm
I20260926 16:34:04.578866 20011 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:34:04 dist-test-slave-khd3 krb5kdc[19998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440444, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:34:04.582062 20011 init.cc:377] Logged in from keytab as kudu/127.6.33.62@KRBTEST.COM (short username kudu)
I20260926 16:34:04.583563 20011 webserver.cc:460] Webserver started at http://127.6.33.62:36925/ using document root <none> and password file <none>
I20260926 16:34:04.584286 20011 fs_manager.cc:359] Metadata directory not provided
I20260926 16:34:04.584435 20011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:34:04.584838 20011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:34:04.587919 20011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440233850748-6276-0/minicluster-data/master-0/data/instance:
uuid: "e2611c33393940b0a6324949b36b54b9"
format_stamp: "Formatted at 2026-09-26 16:34:04 on dist-test-slave-khd3"
server_key: "0c2893ace82af9e03b5f54c5c5e73749"
server_key_iv: "9d77b20182c39679892ae25de8adbf77"
server_key_version: "encryptionkey@0"
I20260926 16:34:04.588691 20011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440233850748-6276-0/minicluster-data/master-0/wal/instance:
uuid: "e2611c33393940b0a6324949b36b54b9"
format_stamp: "Formatted at 2026-09-26 16:34:04 on dist-test-slave-khd3"
server_key: "0c2893ace82af9e03b5f54c5c5e73749"
server_key_iv: "9d77b20182c39679892ae25de8adbf77"
server_key_version: "encryptionkey@0"
I20260926 16:34:04.593983 20011 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 16:34:04.597522 20026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:34:04.598819 20011 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:34:04.598996 20011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440233850748-6276-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440233850748-6276-0/minicluster-data/master-0/wal
uuid: "e2611c33393940b0a6324949b36b54b9"
format_stamp: "Formatted at 2026-09-26 16:34:04 on dist-test-slave-khd3"
server_key: "0c2893ace82af9e03b5f54c5c5e73749"
server_key_iv: "9d77b20182c39679892ae25de8adbf77"
server_key_version: "encryptionkey@0"
I20260926 16:34:04.599203 20011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440233850748-6276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440233850748-6276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440233850748-6276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:34:04.626680 20011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:34:04.631345 20011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:34:04.631675 20011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:34:04.651683 20011 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.62:43961
I20260926 16:34:04.651713 20077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.62:43961 every 8 connection(s)
I20260926 16:34:04.653374 20011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
I20260926 16:34:04.657303 20078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:34:04.659340 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 20011
I20260926 16:34:04.659588 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440233850748-6276-0/minicluster-data/master-0/wal/instance
I20260926 16:34:04.660435 6276 external_mini_cluster.cc:1428] Setting key 2602b986c200d3ca11757eefefcd1d63
I20260926 16:34:04.670737 20078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2611c33393940b0a6324949b36b54b9: Bootstrap starting.
I20260926 16:34:04.674820 20078 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2611c33393940b0a6324949b36b54b9: Neither blocks nor log segments found. Creating new log.
Sep 26 16:34:04 dist-test-slave-khd3 krb5kdc[19998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:34:04.675992 20078 log.cc:824] T 00000000000000000000000000000000 P e2611c33393940b0a6324949b36b54b9: Log is configured to *not* fsync() on all Append() calls
I20260926 16:34:04.679703 20078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2611c33393940b0a6324949b36b54b9: No bootstrap required, opened a new log
I20260926 16:34:04.686049 20081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:04.662507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46800 (local address 127.6.33.62:43961)
0926 16:34:04.663384 (+ 877us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:04.663397 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:04.663441 (+ 44us) server_negotiation.cc:407] Connection header received
0926 16:34:04.665290 (+ 1849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:04.665315 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:04.665741 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:04.666081 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:34:04.667957 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:04.669456 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:04.671237 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:04.671910 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:04.672047 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:04.677300 (+ 5253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:34:04.677308 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:34:04.677323 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:34:04.677396 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:34:04.681282 (+ 3886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:04.682091 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:04.682101 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:04.682111 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:04.682328 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:04.682746 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:04.682751 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:04.682755 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:04.683480 (+ 725us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:34:04.683787 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:04.684170 (+ 383us) server_negotiation.cc:299] Negotiation successful
0926 16:34:04.684592 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":382,"threads_started":1}
I20260926 16:34:04.690107 6276 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:34:04.690111 20078 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2611c33393940b0a6324949b36b54b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2611c33393940b0a6324949b36b54b9" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 43961 } }
I20260926 16:34:04.690527 20078 raft_consensus.cc:374] T 00000000000000000000000000000000 P e2611c33393940b0a6324949b36b54b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:34:04.690630 20078 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2611c33393940b0a6324949b36b54b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2611c33393940b0a6324949b36b54b9, State: Initialized, Role: FOLLOWER
I20260926 16:34:04.691253 20078 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2611c33393940b0a6324949b36b54b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2611c33393940b0a6324949b36b54b9" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 43961 } }
I20260926 16:34:04.691498 20078 raft_consensus.cc:388] T 00000000000000000000000000000000 P e2611c33393940b0a6324949b36b54b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:34:04.691658 20078 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2611c33393940b0a6324949b36b54b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:34:04.691825 20078 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2611c33393940b0a6324949b36b54b9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:34:04.694440 20078 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2611c33393940b0a6324949b36b54b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2611c33393940b0a6324949b36b54b9" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 43961 } }
I20260926 16:34:04.694988 20078 leader_election.cc:302] T 00000000000000000000000000000000 P e2611c33393940b0a6324949b36b54b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e2611c33393940b0a6324949b36b54b9; no voters:
I20260926 16:34:04.695583 20078 leader_election.cc:288] T 00000000000000000000000000000000 P e2611c33393940b0a6324949b36b54b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:34:04.695771 20083 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2611c33393940b0a6324949b36b54b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:34:04.696498 20083 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2611c33393940b0a6324949b36b54b9 [term 1 LEADER]: Becoming Leader. State: Replica: e2611c33393940b0a6324949b36b54b9, State: Running, Role: LEADER
I20260926 16:34:04.697098 20083 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2611c33393940b0a6324949b36b54b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2611c33393940b0a6324949b36b54b9" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 43961 } }
I20260926 16:34:04.697957 20078 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2611c33393940b0a6324949b36b54b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:34:04.700265 20085 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2611c33393940b0a6324949b36b54b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2611c33393940b0a6324949b36b54b9. Latest consensus state: current_term: 1 leader_uuid: "e2611c33393940b0a6324949b36b54b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2611c33393940b0a6324949b36b54b9" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 43961 } } }
I20260926 16:34:04.700309 20084 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2611c33393940b0a6324949b36b54b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e2611c33393940b0a6324949b36b54b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2611c33393940b0a6324949b36b54b9" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 43961 } } }
I20260926 16:34:04.700671 20085 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2611c33393940b0a6324949b36b54b9 [sys.catalog]: This master's current role is: LEADER
I20260926 16:34:04.700730 20084 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2611c33393940b0a6324949b36b54b9 [sys.catalog]: This master's current role is: LEADER
I20260926 16:34:04.702713 20088 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:34:04.708868 20088 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:34:04.718253 20088 catalog_manager.cc:1347] Generated new cluster ID: fb771aecfe634d66a98849512e4bf411
I20260926 16:34:04.718407 20088 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:34:04.739605 20088 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:34:04.741058 20088 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:34:04.756170 20088 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e2611c33393940b0a6324949b36b54b9: Generated new TSK 0
I20260926 16:34:04.757169 20088 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:34:04.802608 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 20011
I20260926 16:34:04.803871 20073 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:34:04.901990 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 20011
2026-09-26T16:34:04Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2054 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:34:05 dist-test-slave-khd3 krb5kdc[20106](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:34:05 dist-test-slave-khd3 krb5kdc[20106](info): set up 2 sockets
Sep 26 16:34:05 dist-test-slave-khd3 krb5kdc[20106](info): commencing operation
krb5kdc: starting...
W20260926 16:34:06.026134 6276 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.008s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Sep 26 16:34:06 dist-test-slave-khd3 krb5kdc[20106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:34:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:34:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab.
Entry for principal kudu/127.6.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab.
I20260926 16:34:06.268643 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:36283
--webserver_interface=127.6.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:44077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.33.62:36283
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:34:06.600494 20119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:34:06.600896 20119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:34:06.601060 20119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:34:06.609176 20119 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:34:06.609335 20119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:34:06.609424 20119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:34:06.609531 20119 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:34:06.609615 20119 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:34:06.619853 20119 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:44077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.33.62:36283
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:36283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:34:06.621378 20119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:34:06.622843 20119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:34:06.636092 20127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:34:06.636135 20124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:34:06.636193 20125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:34:06.637189 20119 server_base.cc:1034] running on GCE node
I20260926 16:34:06.637981 20119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:34:06.639344 20119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:34:06.640570 20119 hybrid_clock.cc:648] HybridClock initialized: now 1790440446640533 us; error 50 us; skew 500 ppm
I20260926 16:34:06.640985 20119 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:34:06 dist-test-slave-khd3 krb5kdc[20106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440446, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:34:06.643966 20119 init.cc:377] Logged in from keytab as kudu/127.6.33.62@KRBTEST.COM (short username kudu)
I20260926 16:34:06.645375 20119 webserver.cc:460] Webserver started at http://127.6.33.62:46143/ using document root <none> and password file <none>
I20260926 16:34:06.645998 20119 fs_manager.cc:359] Metadata directory not provided
I20260926 16:34:06.646096 20119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:34:06.646350 20119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:34:06.649235 20119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/master-0/data/instance:
uuid: "1681ae7327bb4f84a8620eb060da18ce"
format_stamp: "Formatted at 2026-09-26 16:34:06 on dist-test-slave-khd3"
server_key: "31d371d57d2a98fc3e204258b17c14d6"
server_key_iv: "b7c4e0b0d51311bcbb459f9afb5d10a9"
server_key_version: "encryptionkey@0"
I20260926 16:34:06.649914 20119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/master-0/wal/instance:
uuid: "1681ae7327bb4f84a8620eb060da18ce"
format_stamp: "Formatted at 2026-09-26 16:34:06 on dist-test-slave-khd3"
server_key: "31d371d57d2a98fc3e204258b17c14d6"
server_key_iv: "b7c4e0b0d51311bcbb459f9afb5d10a9"
server_key_version: "encryptionkey@0"
I20260926 16:34:06.655019 20119 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 16:34:06.658615 20134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:34:06.659972 20119 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 16:34:06.660109 20119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/master-0/wal
uuid: "1681ae7327bb4f84a8620eb060da18ce"
format_stamp: "Formatted at 2026-09-26 16:34:06 on dist-test-slave-khd3"
server_key: "31d371d57d2a98fc3e204258b17c14d6"
server_key_iv: "b7c4e0b0d51311bcbb459f9afb5d10a9"
server_key_version: "encryptionkey@0"
I20260926 16:34:06.660305 20119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:34:06.677732 20119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:34:06.682148 20119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:34:06.682387 20119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:34:06.701337 20119 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.62:36283
I20260926 16:34:06.701334 20185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.62:36283 every 8 connection(s)
I20260926 16:34:06.702931 20119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
I20260926 16:34:06.706887 20186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:34:06.711825 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 20119
I20260926 16:34:06.712061 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/master-0/wal/instance
I20260926 16:34:06.712725 6276 external_mini_cluster.cc:1428] Setting key 1bf95bff5700b2d6140a68729b563efc
I20260926 16:34:06.718431 20186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1681ae7327bb4f84a8620eb060da18ce: Bootstrap starting.
I20260926 16:34:06.721719 20186 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1681ae7327bb4f84a8620eb060da18ce: Neither blocks nor log segments found. Creating new log.
I20260926 16:34:06.722776 20186 log.cc:824] T 00000000000000000000000000000000 P 1681ae7327bb4f84a8620eb060da18ce: Log is configured to *not* fsync() on all Append() calls
I20260926 16:34:06.726037 20186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1681ae7327bb4f84a8620eb060da18ce: No bootstrap required, opened a new log
Sep 26 16:34:06 dist-test-slave-khd3 krb5kdc[20106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:34:06.734555 20186 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1681ae7327bb4f84a8620eb060da18ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1681ae7327bb4f84a8620eb060da18ce" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 36283 } }
I20260926 16:34:06.735054 20186 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1681ae7327bb4f84a8620eb060da18ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:34:06.735194 20186 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1681ae7327bb4f84a8620eb060da18ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1681ae7327bb4f84a8620eb060da18ce, State: Initialized, Role: FOLLOWER
I20260926 16:34:06.735829 20186 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1681ae7327bb4f84a8620eb060da18ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1681ae7327bb4f84a8620eb060da18ce" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 36283 } }
I20260926 16:34:06.736063 20186 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1681ae7327bb4f84a8620eb060da18ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:34:06.736171 20189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:06.715006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57840 (local address 127.6.33.62:36283)
0926 16:34:06.715955 (+ 949us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:06.715971 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:06.716014 (+ 43us) server_negotiation.cc:407] Connection header received
0926 16:34:06.717802 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:06.717826 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:06.718289 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:06.718711 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:34:06.720203 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:06.721577 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:06.722989 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:06.723726 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:06.723933 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:06.728429 (+ 4496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:34:06.728437 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:34:06.728453 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:34:06.728504 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:34:06.731736 (+ 3232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:06.732449 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:06.732457 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:06.732463 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:06.732647 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:06.733078 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:06.733083 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:06.733089 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:06.733651 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:34:06.733952 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:06.734418 (+ 466us) server_negotiation.cc:299] Negotiation successful
0926 16:34:06.734841 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":720,"thread_start_us":451,"threads_started":1}
I20260926 16:34:06.736232 20186 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1681ae7327bb4f84a8620eb060da18ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:34:06.736423 20186 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1681ae7327bb4f84a8620eb060da18ce [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:34:06.739029 20186 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1681ae7327bb4f84a8620eb060da18ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1681ae7327bb4f84a8620eb060da18ce" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 36283 } }
I20260926 16:34:06.739586 20186 leader_election.cc:302] T 00000000000000000000000000000000 P 1681ae7327bb4f84a8620eb060da18ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1681ae7327bb4f84a8620eb060da18ce; no voters:
I20260926 16:34:06.740182 20186 leader_election.cc:288] T 00000000000000000000000000000000 P 1681ae7327bb4f84a8620eb060da18ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:34:06.740347 20191 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1681ae7327bb4f84a8620eb060da18ce [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:34:06.741026 20191 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1681ae7327bb4f84a8620eb060da18ce [term 1 LEADER]: Becoming Leader. State: Replica: 1681ae7327bb4f84a8620eb060da18ce, State: Running, Role: LEADER
I20260926 16:34:06.741590 20191 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1681ae7327bb4f84a8620eb060da18ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1681ae7327bb4f84a8620eb060da18ce" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 36283 } }
I20260926 16:34:06.742377 20186 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1681ae7327bb4f84a8620eb060da18ce [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:34:06.744544 20192 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1681ae7327bb4f84a8620eb060da18ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1681ae7327bb4f84a8620eb060da18ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1681ae7327bb4f84a8620eb060da18ce" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 36283 } } }
I20260926 16:34:06.744902 20192 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1681ae7327bb4f84a8620eb060da18ce [sys.catalog]: This master's current role is: LEADER
I20260926 16:34:06.744962 20193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1681ae7327bb4f84a8620eb060da18ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1681ae7327bb4f84a8620eb060da18ce. Latest consensus state: current_term: 1 leader_uuid: "1681ae7327bb4f84a8620eb060da18ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1681ae7327bb4f84a8620eb060da18ce" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 36283 } } }
I20260926 16:34:06.745266 20193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1681ae7327bb4f84a8620eb060da18ce [sys.catalog]: This master's current role is: LEADER
I20260926 16:34:06.746878 20198 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:34:06.752424 20198 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:34:06.761426 20198 catalog_manager.cc:1347] Generated new cluster ID: 4a47e9ca2a544b0aa9e2c015c3a18ef4
I20260926 16:34:06.761550 20198 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:34:06.774386 20198 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:34:06.775739 20198 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:34:06.795415 20198 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1681ae7327bb4f84a8620eb060da18ce: Generated new TSK 0
I20260926 16:34:06.796566 20198 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.33.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab.
Entry for principal kudu/127.6.33.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab.
I20260926 16:34:06.858937 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.1:0
--local_ip_for_outbound_sockets=127.6.33.1
--webserver_interface=127.6.33.1
--webserver_port=0
--tserver_master_addrs=127.6.33.62:36283
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab
--principal=kudu/127.6.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:44077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:34:07.194490 20212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:34:07.194831 20212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:34:07.195003 20212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:34:07.202270 20212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:34:07.202435 20212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:34:07.202620 20212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.1
I20260926 16:34:07.212522 20212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:44077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab
--principal=kudu/127.6.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.33.1
--webserver_port=0
--tserver_master_addrs=127.6.33.62:36283
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.1
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:34:07.213894 20212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:34:07.215242 20212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:34:07.228670 20217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:34:07.228717 20218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:34:07.228927 20220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:34:07.229651 20212 server_base.cc:1034] running on GCE node
I20260926 16:34:07.230347 20212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:34:07.231417 20212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:34:07.232632 20212 hybrid_clock.cc:648] HybridClock initialized: now 1790440447232584 us; error 41 us; skew 500 ppm
I20260926 16:34:07.233012 20212 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:34:07 dist-test-slave-khd3 krb5kdc[20106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440447, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:34:07.236420 20212 init.cc:377] Logged in from keytab as kudu/127.6.33.1@KRBTEST.COM (short username kudu)
I20260926 16:34:07.238152 20212 webserver.cc:460] Webserver started at http://127.6.33.1:41611/ using document root <none> and password file <none>
I20260926 16:34:07.238919 20212 fs_manager.cc:359] Metadata directory not provided
I20260926 16:34:07.239024 20212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:34:07.239283 20212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:34:07.242631 20212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-0/data/instance:
uuid: "fa9aad2f0e384189872e794e35bbad29"
format_stamp: "Formatted at 2026-09-26 16:34:07 on dist-test-slave-khd3"
server_key: "0a324ffecaed301b7df470ea749a8246"
server_key_iv: "9e692dee5188eb253425075ecc5d2f6c"
server_key_version: "encryptionkey@0"
I20260926 16:34:07.243484 20212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-0/wal/instance:
uuid: "fa9aad2f0e384189872e794e35bbad29"
format_stamp: "Formatted at 2026-09-26 16:34:07 on dist-test-slave-khd3"
server_key: "0a324ffecaed301b7df470ea749a8246"
server_key_iv: "9e692dee5188eb253425075ecc5d2f6c"
server_key_version: "encryptionkey@0"
I20260926 16:34:07.249127 20212 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.007s
I20260926 16:34:07.252562 20227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:34:07.254020 20212 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:34:07.254189 20212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-0/wal
uuid: "fa9aad2f0e384189872e794e35bbad29"
format_stamp: "Formatted at 2026-09-26 16:34:07 on dist-test-slave-khd3"
server_key: "0a324ffecaed301b7df470ea749a8246"
server_key_iv: "9e692dee5188eb253425075ecc5d2f6c"
server_key_version: "encryptionkey@0"
I20260926 16:34:07.254371 20212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:34:07.285076 20212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:34:07.290081 20212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:34:07.290406 20212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:34:07.291450 20212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:34:07.292917 20212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:34:07.293002 20212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:34:07.293164 20212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:34:07.293231 20212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:34:07.326931 20212 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.1:34095
I20260926 16:34:07.326987 20339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.1:34095 every 8 connection(s)
I20260926 16:34:07.328248 20212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
I20260926 16:34:07.334790 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 20212
I20260926 16:34:07.334996 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-0/wal/instance
I20260926 16:34:07.335605 6276 external_mini_cluster.cc:1428] Setting key 201865d4e0c71a3157de5ac05eb0a86c
Sep 26 16:34:07 dist-test-slave-khd3 krb5kdc[20106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440447, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.1@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:34:07.349432 20342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:07.330863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:43651 (local address 127.6.33.62:36283)
0926 16:34:07.331359 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:07.331366 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:07.332289 (+ 923us) server_negotiation.cc:407] Connection header received
0926 16:34:07.334141 (+ 1852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:07.334145 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:07.334216 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:07.334311 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:34:07.336010 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:07.337028 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:07.338849 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:07.339614 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:07.339767 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:07.343359 (+ 3592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:34:07.343368 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:34:07.343376 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:34:07.343421 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:34:07.346340 (+ 2919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:07.347084 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:07.347090 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:07.347094 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:07.347223 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:07.347676 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:07.347682 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:07.347686 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:07.347946 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:34:07.348114 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:07.348840 (+ 726us) server_negotiation.cc:299] Negotiation successful
0926 16:34:07.349065 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":239,"threads_started":1}
I20260926 16:34:07.350418 20343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:07.331397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36283 (local address 127.6.33.1:43651)
0926 16:34:07.332126 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:07.332166 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 16:34:07.333884 (+ 1718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:34:07.334523 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:34:07.334531 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:34:07.335183 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:34:07.335824 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:07.335843 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:07.337194 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:07.337199 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:07.338694 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:07.338706 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:07.339822 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:07.339826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:07.340028 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:07.340725 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:34:07.340749 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:34:07.343155 (+ 2406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:34:07.346504 (+ 3349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:34:07.346513 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:34:07.346530 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:34:07.346959 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:34:07.347336 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:34:07.347341 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:34:07.347344 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:34:07.347565 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:34:07.348076 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:34:07.348083 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:34:07.348440 (+ 357us) client_negotiation.cc:769] Sending connection context
0926 16:34:07.348713 (+ 273us) client_negotiation.cc:240] Negotiation successful
0926 16:34:07.349266 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":302,"threads_started":1}
I20260926 16:34:07.352090 20340 heartbeater.cc:347] Connected to a master server at 127.6.33.62:36283
I20260926 16:34:07.352421 20340 heartbeater.cc:464] Registering TS with master...
I20260926 16:34:07.353180 20340 heartbeater.cc:511] Master 127.6.33.62:36283 requested a full tablet report, sending...
I20260926 16:34:07.355590 20151 ts_manager.cc:194] Registered new tserver with Master: fa9aad2f0e384189872e794e35bbad29 (127.6.33.1:34095)
I20260926 16:34:07.357813 20151 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.33.1@KRBTEST.COM'} at 127.6.33.1:43651
WARNING: no policy specified for kudu/127.6.33.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab.
Entry for principal kudu/127.6.33.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab.
I20260926 16:34:07.387877 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.2:0
--local_ip_for_outbound_sockets=127.6.33.2
--webserver_interface=127.6.33.2
--webserver_port=0
--tserver_master_addrs=127.6.33.62:36283
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab
--principal=kudu/127.6.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:44077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:34:07.727957 20346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:34:07.728308 20346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:34:07.728484 20346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:34:07.736631 20346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:34:07.736826 20346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:34:07.737028 20346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.2
I20260926 16:34:07.747973 20346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:44077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab
--principal=kudu/127.6.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.33.2
--webserver_port=0
--tserver_master_addrs=127.6.33.62:36283
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.2
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:34:07.749462 20346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:34:07.750808 20346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:34:07.764406 20351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:34:07.764384 20352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:34:07.764580 20354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:34:07.765261 20346 server_base.cc:1034] running on GCE node
I20260926 16:34:07.765868 20346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:34:07.766853 20346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:34:07.768091 20346 hybrid_clock.cc:648] HybridClock initialized: now 1790440447768046 us; error 33 us; skew 500 ppm
I20260926 16:34:07.768536 20346 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:34:07 dist-test-slave-khd3 krb5kdc[20106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440447, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:34:07.771828 20346 init.cc:377] Logged in from keytab as kudu/127.6.33.2@KRBTEST.COM (short username kudu)
I20260926 16:34:07.773401 20346 webserver.cc:460] Webserver started at http://127.6.33.2:38415/ using document root <none> and password file <none>
I20260926 16:34:07.774073 20346 fs_manager.cc:359] Metadata directory not provided
I20260926 16:34:07.774174 20346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:34:07.774461 20346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:34:07.777454 20346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-1/data/instance:
uuid: "c6ed5f76dd9e4a4593e96a1eaad01553"
format_stamp: "Formatted at 2026-09-26 16:34:07 on dist-test-slave-khd3"
server_key: "cd5beecea99a70c7b4a489beb613869e"
server_key_iv: "5a67ea834ccffb23601ab2c028fdf783"
server_key_version: "encryptionkey@0"
I20260926 16:34:07.778175 20346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-1/wal/instance:
uuid: "c6ed5f76dd9e4a4593e96a1eaad01553"
format_stamp: "Formatted at 2026-09-26 16:34:07 on dist-test-slave-khd3"
server_key: "cd5beecea99a70c7b4a489beb613869e"
server_key_iv: "5a67ea834ccffb23601ab2c028fdf783"
server_key_version: "encryptionkey@0"
I20260926 16:34:07.783401 20346 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 16:34:07.786945 20361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:34:07.788360 20346 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 16:34:07.788548 20346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-1/wal
uuid: "c6ed5f76dd9e4a4593e96a1eaad01553"
format_stamp: "Formatted at 2026-09-26 16:34:07 on dist-test-slave-khd3"
server_key: "cd5beecea99a70c7b4a489beb613869e"
server_key_iv: "5a67ea834ccffb23601ab2c028fdf783"
server_key_version: "encryptionkey@0"
I20260926 16:34:07.788727 20346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:34:07.807505 20346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:34:07.812057 20346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:34:07.812368 20346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:34:07.813390 20346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:34:07.814723 20346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:34:07.814795 20346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:34:07.814965 20346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:34:07.815017 20346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:34:07.848223 20346 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.2:33857
I20260926 16:34:07.848248 20473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.2:33857 every 8 connection(s)
I20260926 16:34:07.849646 20346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
I20260926 16:34:07.850136 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 20346
I20260926 16:34:07.850324 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-1/wal/instance
I20260926 16:34:07.850944 6276 external_mini_cluster.cc:1428] Setting key e771c4e483b05aed9e8ea3949c39acb4
Sep 26 16:34:07 dist-test-slave-khd3 krb5kdc[20106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440447, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.2@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:34:07.871703 20476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:07.852448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:48727 (local address 127.6.33.62:36283)
0926 16:34:07.852951 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:07.852957 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:07.854145 (+ 1188us) server_negotiation.cc:407] Connection header received
0926 16:34:07.855972 (+ 1827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:07.855980 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:07.856071 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:07.856200 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:34:07.857778 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:07.859043 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:07.861099 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:07.861482 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:07.861581 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:07.865882 (+ 4301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:34:07.865887 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:34:07.865891 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:34:07.865923 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:34:07.868558 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:07.869326 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:07.869331 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:07.869335 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:07.869467 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:07.869915 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:07.869920 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:07.869924 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:07.870208 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:34:07.870376 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:07.871083 (+ 707us) server_negotiation.cc:299] Negotiation successful
0926 16:34:07.871322 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":224,"threads_started":1}
I20260926 16:34:07.872699 20477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:07.853150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36283 (local address 127.6.33.2:48727)
0926 16:34:07.853969 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:07.854015 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 16:34:07.855707 (+ 1692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:34:07.856398 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:34:07.856405 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:34:07.856994 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:34:07.857605 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:07.857630 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:07.859197 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:07.859201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:07.860962 (+ 1761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:07.860979 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:07.861704 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:07.861714 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:07.862031 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:07.862778 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:34:07.862817 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:34:07.865633 (+ 2816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:34:07.868699 (+ 3066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:34:07.868708 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:34:07.868725 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:34:07.869198 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:34:07.869580 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:34:07.869585 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:34:07.869596 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:34:07.869794 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:34:07.870337 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:34:07.870345 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:34:07.870704 (+ 359us) client_negotiation.cc:769] Sending connection context
0926 16:34:07.870987 (+ 283us) client_negotiation.cc:240] Negotiation successful
0926 16:34:07.871536 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":326,"threads_started":1}
WARNING: no policy specified for kudu/127.6.33.3@KRBTEST.COM; defaulting to no policy
I20260926 16:34:07.874233 20474 heartbeater.cc:347] Connected to a master server at 127.6.33.62:36283
I20260926 16:34:07.874588 20474 heartbeater.cc:464] Registering TS with master...
I20260926 16:34:07.875367 20474 heartbeater.cc:511] Master 127.6.33.62:36283 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.3@KRBTEST.COM" created.
I20260926 16:34:07.876991 20151 ts_manager.cc:194] Registered new tserver with Master: c6ed5f76dd9e4a4593e96a1eaad01553 (127.6.33.2:33857)
I20260926 16:34:07.877804 20151 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.33.2@KRBTEST.COM'} at 127.6.33.2:48727
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab.
Entry for principal kudu/127.6.33.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab.
I20260926 16:34:07.904210 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.3:0
--local_ip_for_outbound_sockets=127.6.33.3
--webserver_interface=127.6.33.3
--webserver_port=0
--tserver_master_addrs=127.6.33.62:36283
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab
--principal=kudu/127.6.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:44077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:34:08.235791 20480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:34:08.236076 20480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:34:08.236274 20480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:34:08.243862 20480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:34:08.244006 20480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:34:08.244175 20480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.3
I20260926 16:34:08.254881 20480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:44077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab
--principal=kudu/127.6.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.33.3
--webserver_port=0
--tserver_master_addrs=127.6.33.62:36283
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.3
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:34:08.256209 20480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:34:08.257743 20480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:34:08.271121 20485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:34:08.271195 20486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:34:08.271217 20488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:34:08.271862 20480 server_base.cc:1034] running on GCE node
I20260926 16:34:08.272451 20480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:34:08.273450 20480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:34:08.274765 20480 hybrid_clock.cc:648] HybridClock initialized: now 1790440448274723 us; error 67 us; skew 500 ppm
I20260926 16:34:08.275146 20480 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:34:08 dist-test-slave-khd3 krb5kdc[20106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440448, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:34:08.278483 20480 init.cc:377] Logged in from keytab as kudu/127.6.33.3@KRBTEST.COM (short username kudu)
I20260926 16:34:08.280023 20480 webserver.cc:460] Webserver started at http://127.6.33.3:36299/ using document root <none> and password file <none>
I20260926 16:34:08.280841 20480 fs_manager.cc:359] Metadata directory not provided
I20260926 16:34:08.281032 20480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:34:08.281363 20480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:34:08.284451 20480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-2/data/instance:
uuid: "ca053d21d9304c0aac25ce7dc6fdacf6"
format_stamp: "Formatted at 2026-09-26 16:34:08 on dist-test-slave-khd3"
server_key: "95608f2856ca05c3a623b28ff261dea6"
server_key_iv: "3cebd328b8ea9a153efe2e27eccbd748"
server_key_version: "encryptionkey@0"
I20260926 16:34:08.285375 20480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-2/wal/instance:
uuid: "ca053d21d9304c0aac25ce7dc6fdacf6"
format_stamp: "Formatted at 2026-09-26 16:34:08 on dist-test-slave-khd3"
server_key: "95608f2856ca05c3a623b28ff261dea6"
server_key_iv: "3cebd328b8ea9a153efe2e27eccbd748"
server_key_version: "encryptionkey@0"
I20260926 16:34:08.291164 20480 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 16:34:08.295562 20495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:34:08.297061 20480 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:34:08.297251 20480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-2/wal
uuid: "ca053d21d9304c0aac25ce7dc6fdacf6"
format_stamp: "Formatted at 2026-09-26 16:34:08 on dist-test-slave-khd3"
server_key: "95608f2856ca05c3a623b28ff261dea6"
server_key_iv: "3cebd328b8ea9a153efe2e27eccbd748"
server_key_version: "encryptionkey@0"
I20260926 16:34:08.297464 20480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:34:08.310110 20480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:34:08.315047 20480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:34:08.315431 20480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:34:08.316480 20480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:34:08.317997 20480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:34:08.318121 20480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:34:08.318274 20480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:34:08.318346 20480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:34:08.352916 20480 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.3:45107
I20260926 16:34:08.352923 20607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.3:45107 every 8 connection(s)
I20260926 16:34:08.354319 20480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
I20260926 16:34:08.356666 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 20480
I20260926 16:34:08.356877 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-2/wal/instance
I20260926 16:34:08.357425 6276 external_mini_cluster.cc:1428] Setting key bf4aa5027ce02fe98c0998a5d84bf48c
I20260926 16:34:08.360903 20340 heartbeater.cc:503] Master 127.6.33.62:36283 was elected leader, sending a full tablet report...
Sep 26 16:34:08 dist-test-slave-khd3 krb5kdc[20106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440448, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.3@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:34:08.375279 20476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:08.357033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.3:55969 (local address 127.6.33.62:36283)
0926 16:34:08.357212 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:08.357217 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:08.358652 (+ 1435us) server_negotiation.cc:407] Connection header received
0926 16:34:08.361219 (+ 2567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:08.361224 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:08.361307 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:08.361434 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:34:08.362868 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:08.363662 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:08.365314 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:08.365731 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:08.365868 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:08.369304 (+ 3436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:34:08.369310 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:34:08.369321 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:34:08.369360 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:34:08.372204 (+ 2844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:08.373176 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:08.373181 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:08.373185 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:08.373316 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:08.373724 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:08.373731 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:08.373737 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:08.374074 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:34:08.374197 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:08.374866 (+ 669us) server_negotiation.cc:299] Negotiation successful
0926 16:34:08.375035 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 16:34:08.376276 20610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:08.357567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36283 (local address 127.6.33.3:55969)
0926 16:34:08.358466 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:08.358532 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0926 16:34:08.360923 (+ 2391us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:34:08.361661 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:34:08.361669 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:34:08.362163 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:34:08.362707 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:08.362724 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:08.363828 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:08.363833 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:08.365168 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:08.365180 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:08.365955 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:08.365962 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:08.366203 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:08.366833 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:34:08.366856 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:34:08.369099 (+ 2243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:34:08.372347 (+ 3248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:34:08.372354 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:34:08.372368 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:34:08.373035 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:34:08.373409 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:34:08.373412 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:34:08.373415 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:34:08.373589 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:34:08.374194 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:34:08.374200 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:34:08.374516 (+ 316us) client_negotiation.cc:769] Sending connection context
0926 16:34:08.374757 (+ 241us) client_negotiation.cc:240] Negotiation successful
0926 16:34:08.375142 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":359,"threads_started":1}
I20260926 16:34:08.377674 20608 heartbeater.cc:347] Connected to a master server at 127.6.33.62:36283
I20260926 16:34:08.378000 20608 heartbeater.cc:464] Registering TS with master...
I20260926 16:34:08.378646 20608 heartbeater.cc:511] Master 127.6.33.62:36283 requested a full tablet report, sending...
I20260926 16:34:08.379941 20151 ts_manager.cc:194] Registered new tserver with Master: ca053d21d9304c0aac25ce7dc6fdacf6 (127.6.33.3:45107)
I20260926 16:34:08.380738 20151 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.33.3@KRBTEST.COM'} at 127.6.33.3:55969
I20260926 16:34:08.385262 6276 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:34:08.401510 20476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:08.389522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57846 (local address 127.6.33.62:36283)
0926 16:34:08.389755 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:08.389764 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:08.390096 (+ 332us) server_negotiation.cc:407] Connection header received
0926 16:34:08.390231 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:08.390234 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:08.390294 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:08.390373 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:34:08.391655 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:08.392449 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:08.393828 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:08.394219 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:08.394312 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:08.396646 (+ 2334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:34:08.396650 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:34:08.396654 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:34:08.396682 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:34:08.399043 (+ 2361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:08.399622 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:08.399626 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:08.399628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:08.399739 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:08.400033 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:08.400036 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:08.400038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:08.400349 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:34:08.400561 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:08.400965 (+ 404us) server_negotiation.cc:299] Negotiation successful
0926 16:34:08.401168 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 16:34:08.405687 20151 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57846:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:34:08.412621 20151 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:34:08.437659 20622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:08.424801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:34095 (local address 127.0.0.1:53546)
0926 16:34:08.427180 (+ 2379us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:08.427205 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 16:34:08.427338 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:34:08.428829 (+ 1491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:34:08.428841 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:34:08.428887 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:34:08.429472 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:08.429494 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:08.432309 (+ 2815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:08.432314 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:08.434562 (+ 2248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:08.434574 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:08.436523 (+ 1949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:08.436533 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:08.436743 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:08.436841 (+ 98us) client_negotiation.cc:769] Sending connection context
0926 16:34:08.436997 (+ 156us) client_negotiation.cc:240] Negotiation successful
0926 16:34:08.437225 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2233,"mutex_wait_us":3,"thread_start_us":325,"threads_started":1}
I20260926 16:34:08.438088 20621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:08.425046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44676 (local address 127.6.33.3:45107)
0926 16:34:08.428890 (+ 3844us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:08.428896 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:08.428923 (+ 27us) server_negotiation.cc:407] Connection header received
0926 16:34:08.429004 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:08.429009 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:08.429194 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:08.429408 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:34:08.430383 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:08.431697 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:08.434806 (+ 3109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:08.436300 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:08.436437 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:08.436696 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:08.437293 (+ 597us) server_negotiation.cc:299] Negotiation successful
0926 16:34:08.437457 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3650,"thread_start_us":3482,"threads_started":1}
I20260926 16:34:08.438205 20623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:08.425367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53546 (local address 127.6.33.1:34095)
0926 16:34:08.427996 (+ 2629us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:08.428008 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:08.428045 (+ 37us) server_negotiation.cc:407] Connection header received
0926 16:34:08.428180 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:08.428191 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:08.428534 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:08.429762 (+ 1228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:34:08.430073 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:08.432181 (+ 2108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:08.434728 (+ 2547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:08.436300 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:08.436437 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:08.436697 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:08.437093 (+ 396us) server_negotiation.cc:299] Negotiation successful
0926 16:34:08.437301 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2410,"thread_start_us":2250,"threads_started":1}
I20260926 16:34:08.439169 20620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:08.424570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.3:45107 (local address 127.0.0.1:44676)
0926 16:34:08.425461 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:08.425503 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 16:34:08.425745 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:34:08.429389 (+ 3644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:34:08.429398 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:34:08.429425 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:34:08.429808 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:08.429822 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:08.431846 (+ 2024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:08.431853 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:08.434579 (+ 2726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:08.434594 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:08.436888 (+ 2294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:08.436894 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:08.437082 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:08.437116 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 16:34:08.438579 (+ 1463us) client_negotiation.cc:240] Negotiation successful
0926 16:34:08.438728 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"mutex_wait_us":28,"thread_start_us":3057,"threads_started":1}
I20260926 16:34:08.439993 20625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:08.427096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.2:33857 (local address 127.0.0.1:60898)
0926 16:34:08.428477 (+ 1381us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:08.428500 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 16:34:08.428630 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:34:08.433208 (+ 4578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:34:08.433217 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:34:08.433244 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:34:08.433749 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:08.433762 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:08.435403 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:08.435410 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:08.436961 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:08.436976 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:08.439209 (+ 2233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:08.439215 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:08.439392 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:08.439427 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 16:34:08.439530 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 16:34:08.439678 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1249,"thread_start_us":1086,"threads_started":1}
I20260926 16:34:08.440132 20624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:08.427111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60898 (local address 127.6.33.2:33857)
0926 16:34:08.432304 (+ 5193us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:08.432312 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:08.432338 (+ 26us) server_negotiation.cc:407] Connection header received
0926 16:34:08.432436 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:08.432443 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:08.432738 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:08.432964 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:34:08.433954 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:08.435265 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:08.437151 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:08.438723 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:08.438842 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:08.439034 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:08.439594 (+ 560us) server_negotiation.cc:299] Negotiation successful
0926 16:34:08.439803 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5022,"thread_start_us":4900,"threads_started":1}
I20260926 16:34:08.442332 20275 tablet_service.cc:1467] Processing CreateTablet for tablet f5c00121e5bb4e88b3a5e14392936713 (DEFAULT_TABLE table=test-table [id=b8c6bd2f0d7948b38607bca8f4ba39cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:34:08.442485 20543 tablet_service.cc:1467] Processing CreateTablet for tablet f5c00121e5bb4e88b3a5e14392936713 (DEFAULT_TABLE table=test-table [id=b8c6bd2f0d7948b38607bca8f4ba39cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:34:08.443919 20543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5c00121e5bb4e88b3a5e14392936713. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:34:08.443928 20275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5c00121e5bb4e88b3a5e14392936713. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:34:08.446661 20409 tablet_service.cc:1467] Processing CreateTablet for tablet f5c00121e5bb4e88b3a5e14392936713 (DEFAULT_TABLE table=test-table [id=b8c6bd2f0d7948b38607bca8f4ba39cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:34:08.448107 20409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5c00121e5bb4e88b3a5e14392936713. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:34:08.455796 20626 tablet_bootstrap.cc:492] T f5c00121e5bb4e88b3a5e14392936713 P fa9aad2f0e384189872e794e35bbad29: Bootstrap starting.
I20260926 16:34:08.459192 20627 tablet_bootstrap.cc:492] T f5c00121e5bb4e88b3a5e14392936713 P ca053d21d9304c0aac25ce7dc6fdacf6: Bootstrap starting.
I20260926 16:34:08.460250 20626 tablet_bootstrap.cc:654] T f5c00121e5bb4e88b3a5e14392936713 P fa9aad2f0e384189872e794e35bbad29: Neither blocks nor log segments found. Creating new log.
I20260926 16:34:08.461683 20626 log.cc:824] T f5c00121e5bb4e88b3a5e14392936713 P fa9aad2f0e384189872e794e35bbad29: Log is configured to *not* fsync() on all Append() calls
I20260926 16:34:08.462673 20628 tablet_bootstrap.cc:492] T f5c00121e5bb4e88b3a5e14392936713 P c6ed5f76dd9e4a4593e96a1eaad01553: Bootstrap starting.
I20260926 16:34:08.463133 20627 tablet_bootstrap.cc:654] T f5c00121e5bb4e88b3a5e14392936713 P ca053d21d9304c0aac25ce7dc6fdacf6: Neither blocks nor log segments found. Creating new log.
I20260926 16:34:08.464943 20627 log.cc:824] T f5c00121e5bb4e88b3a5e14392936713 P ca053d21d9304c0aac25ce7dc6fdacf6: Log is configured to *not* fsync() on all Append() calls
I20260926 16:34:08.466499 20628 tablet_bootstrap.cc:654] T f5c00121e5bb4e88b3a5e14392936713 P c6ed5f76dd9e4a4593e96a1eaad01553: Neither blocks nor log segments found. Creating new log.
I20260926 16:34:08.467785 20628 log.cc:824] T f5c00121e5bb4e88b3a5e14392936713 P c6ed5f76dd9e4a4593e96a1eaad01553: Log is configured to *not* fsync() on all Append() calls
I20260926 16:34:08.469010 20627 tablet_bootstrap.cc:492] T f5c00121e5bb4e88b3a5e14392936713 P ca053d21d9304c0aac25ce7dc6fdacf6: No bootstrap required, opened a new log
I20260926 16:34:08.469089 20626 tablet_bootstrap.cc:492] T f5c00121e5bb4e88b3a5e14392936713 P fa9aad2f0e384189872e794e35bbad29: No bootstrap required, opened a new log
I20260926 16:34:08.469323 20627 ts_tablet_manager.cc:1397] T f5c00121e5bb4e88b3a5e14392936713 P ca053d21d9304c0aac25ce7dc6fdacf6: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260926 16:34:08.469388 20626 ts_tablet_manager.cc:1397] T f5c00121e5bb4e88b3a5e14392936713 P fa9aad2f0e384189872e794e35bbad29: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.005s
I20260926 16:34:08.470790 20628 tablet_bootstrap.cc:492] T f5c00121e5bb4e88b3a5e14392936713 P c6ed5f76dd9e4a4593e96a1eaad01553: No bootstrap required, opened a new log
I20260926 16:34:08.471074 20628 ts_tablet_manager.cc:1397] T f5c00121e5bb4e88b3a5e14392936713 P c6ed5f76dd9e4a4593e96a1eaad01553: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 16:34:08.476521 20627 raft_consensus.cc:348] T f5c00121e5bb4e88b3a5e14392936713 P ca053d21d9304c0aac25ce7dc6fdacf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca053d21d9304c0aac25ce7dc6fdacf6" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45107 } } peers { permanent_uuid: "fa9aad2f0e384189872e794e35bbad29" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 34095 } } peers { permanent_uuid: "c6ed5f76dd9e4a4593e96a1eaad01553" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 33857 } }
I20260926 16:34:08.477077 20627 raft_consensus.cc:374] T f5c00121e5bb4e88b3a5e14392936713 P ca053d21d9304c0aac25ce7dc6fdacf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:34:08.477221 20627 raft_consensus.cc:729] T f5c00121e5bb4e88b3a5e14392936713 P ca053d21d9304c0aac25ce7dc6fdacf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca053d21d9304c0aac25ce7dc6fdacf6, State: Initialized, Role: FOLLOWER
I20260926 16:34:08.477947 20627 consensus_queue.cc:260] T f5c00121e5bb4e88b3a5e14392936713 P ca053d21d9304c0aac25ce7dc6fdacf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca053d21d9304c0aac25ce7dc6fdacf6" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45107 } } peers { permanent_uuid: "fa9aad2f0e384189872e794e35bbad29" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 34095 } } peers { permanent_uuid: "c6ed5f76dd9e4a4593e96a1eaad01553" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 33857 } }
I20260926 16:34:08.479574 20608 heartbeater.cc:503] Master 127.6.33.62:36283 was elected leader, sending a full tablet report...
I20260926 16:34:08.480422 20627 ts_tablet_manager.cc:1428] T f5c00121e5bb4e88b3a5e14392936713 P ca053d21d9304c0aac25ce7dc6fdacf6: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260926 16:34:08.480300 20626 raft_consensus.cc:348] T f5c00121e5bb4e88b3a5e14392936713 P fa9aad2f0e384189872e794e35bbad29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca053d21d9304c0aac25ce7dc6fdacf6" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45107 } } peers { permanent_uuid: "fa9aad2f0e384189872e794e35bbad29" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 34095 } } peers { permanent_uuid: "c6ed5f76dd9e4a4593e96a1eaad01553" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 33857 } }
I20260926 16:34:08.480790 20626 raft_consensus.cc:374] T f5c00121e5bb4e88b3a5e14392936713 P fa9aad2f0e384189872e794e35bbad29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:34:08.480891 20626 raft_consensus.cc:729] T f5c00121e5bb4e88b3a5e14392936713 P fa9aad2f0e384189872e794e35bbad29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa9aad2f0e384189872e794e35bbad29, State: Initialized, Role: FOLLOWER
I20260926 16:34:08.481664 20626 consensus_queue.cc:260] T f5c00121e5bb4e88b3a5e14392936713 P fa9aad2f0e384189872e794e35bbad29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca053d21d9304c0aac25ce7dc6fdacf6" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45107 } } peers { permanent_uuid: "fa9aad2f0e384189872e794e35bbad29" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 34095 } } peers { permanent_uuid: "c6ed5f76dd9e4a4593e96a1eaad01553" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 33857 } }
I20260926 16:34:08.481804 20628 raft_consensus.cc:348] T f5c00121e5bb4e88b3a5e14392936713 P c6ed5f76dd9e4a4593e96a1eaad01553 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca053d21d9304c0aac25ce7dc6fdacf6" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45107 } } peers { permanent_uuid: "fa9aad2f0e384189872e794e35bbad29" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 34095 } } peers { permanent_uuid: "c6ed5f76dd9e4a4593e96a1eaad01553" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 33857 } }
I20260926 16:34:08.482366 20628 raft_consensus.cc:374] T f5c00121e5bb4e88b3a5e14392936713 P c6ed5f76dd9e4a4593e96a1eaad01553 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:34:08.482506 20628 raft_consensus.cc:729] T f5c00121e5bb4e88b3a5e14392936713 P c6ed5f76dd9e4a4593e96a1eaad01553 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ed5f76dd9e4a4593e96a1eaad01553, State: Initialized, Role: FOLLOWER
I20260926 16:34:08.483158 20628 consensus_queue.cc:260] T f5c00121e5bb4e88b3a5e14392936713 P c6ed5f76dd9e4a4593e96a1eaad01553 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca053d21d9304c0aac25ce7dc6fdacf6" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45107 } } peers { permanent_uuid: "fa9aad2f0e384189872e794e35bbad29" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 34095 } } peers { permanent_uuid: "c6ed5f76dd9e4a4593e96a1eaad01553" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 33857 } }
I20260926 16:34:08.483582 20626 ts_tablet_manager.cc:1428] T f5c00121e5bb4e88b3a5e14392936713 P fa9aad2f0e384189872e794e35bbad29: Time spent starting tablet: real 0.014s user 0.010s sys 0.003s
I20260926 16:34:08.484659 20474 heartbeater.cc:503] Master 127.6.33.62:36283 was elected leader, sending a full tablet report...
I20260926 16:34:08.485348 20628 ts_tablet_manager.cc:1428] T f5c00121e5bb4e88b3a5e14392936713 P c6ed5f76dd9e4a4593e96a1eaad01553: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260926 16:34:08.522534 20632 raft_consensus.cc:482] T f5c00121e5bb4e88b3a5e14392936713 P ca053d21d9304c0aac25ce7dc6fdacf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:34:08.522820 20632 raft_consensus.cc:504] T f5c00121e5bb4e88b3a5e14392936713 P ca053d21d9304c0aac25ce7dc6fdacf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca053d21d9304c0aac25ce7dc6fdacf6" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45107 } } peers { permanent_uuid: "fa9aad2f0e384189872e794e35bbad29" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 34095 } } peers { permanent_uuid: "c6ed5f76dd9e4a4593e96a1eaad01553" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 33857 } }
I20260926 16:34:08.524533 20632 leader_election.cc:288] T f5c00121e5bb4e88b3a5e14392936713 P ca053d21d9304c0aac25ce7dc6fdacf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa9aad2f0e384189872e794e35bbad29 (127.6.33.1:34095), c6ed5f76dd9e4a4593e96a1eaad01553 (127.6.33.2:33857)
I20260926 16:34:08.531263 20610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:08.524892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.2:33857 (local address 127.6.33.3:35199)
0926 16:34:08.525217 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:08.525237 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 16:34:08.525382 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:34:08.525772 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:34:08.525778 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:34:08.525812 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:34:08.526042 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:08.526056 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:08.527347 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:08.527352 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:08.529411 (+ 2059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:08.529426 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:08.530706 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:08.530710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:08.530842 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:08.530867 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 16:34:08.530941 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 16:34:08.531039 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":1}
I20260926 16:34:08.531411 20624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:08.525184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.3:35199 (local address 127.6.33.2:33857)
0926 16:34:08.525425 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:08.525434 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:08.525458 (+ 24us) server_negotiation.cc:407] Connection header received
0926 16:34:08.525539 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:08.525545 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:08.525630 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:08.525739 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:34:08.526217 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:08.527193 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:08.529568 (+ 2375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:08.530548 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:08.530685 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:08.530793 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:08.530993 (+ 200us) server_negotiation.cc:299] Negotiation successful
0926 16:34:08.531139 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 16:34:08.532301 20623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:08.525007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.3:56345 (local address 127.6.33.1:34095)
0926 16:34:08.525219 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:08.525227 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:08.525987 (+ 760us) server_negotiation.cc:407] Connection header received
0926 16:34:08.526192 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:08.526197 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:08.526302 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:08.526389 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:34:08.526979 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:08.527949 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:08.530125 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:08.531326 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:08.531481 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:08.531600 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:08.532014 (+ 414us) server_negotiation.cc:299] Negotiation successful
0926 16:34:08.532128 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 16:34:08.532491 20635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:08.524953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:34095 (local address 127.6.33.3:56345)
0926 16:34:08.525877 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:08.525894 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 16:34:08.526014 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:34:08.526430 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:34:08.526434 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:34:08.526459 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:34:08.526801 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:08.526813 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:08.528100 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:08.528105 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:08.529972 (+ 1867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:08.529983 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:08.531712 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:08.531716 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:08.531829 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:08.531852 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 16:34:08.531961 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 16:34:08.532118 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":806,"mutex_wait_us":28,"thread_start_us":454,"threads_started":1}
I20260926 16:34:08.532485 20429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c00121e5bb4e88b3a5e14392936713" candidate_uuid: "ca053d21d9304c0aac25ce7dc6fdacf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ed5f76dd9e4a4593e96a1eaad01553" is_pre_election: true
I20260926 16:34:08.533166 20429 raft_consensus.cc:2454] T f5c00121e5bb4e88b3a5e14392936713 P c6ed5f76dd9e4a4593e96a1eaad01553 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca053d21d9304c0aac25ce7dc6fdacf6 in term 0.
I20260926 16:34:08.533717 20295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c00121e5bb4e88b3a5e14392936713" candidate_uuid: "ca053d21d9304c0aac25ce7dc6fdacf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa9aad2f0e384189872e794e35bbad29" is_pre_election: true
I20260926 16:34:08.533921 20496 leader_election.cc:302] T f5c00121e5bb4e88b3a5e14392936713 P ca053d21d9304c0aac25ce7dc6fdacf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6ed5f76dd9e4a4593e96a1eaad01553, ca053d21d9304c0aac25ce7dc6fdacf6; no voters:
I20260926 16:34:08.534289 20295 raft_consensus.cc:2454] T f5c00121e5bb4e88b3a5e14392936713 P fa9aad2f0e384189872e794e35bbad29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca053d21d9304c0aac25ce7dc6fdacf6 in term 0.
I20260926 16:34:08.534363 20632 raft_consensus.cc:2781] T f5c00121e5bb4e88b3a5e14392936713 P ca053d21d9304c0aac25ce7dc6fdacf6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:34:08.534507 20632 raft_consensus.cc:482] T f5c00121e5bb4e88b3a5e14392936713 P ca053d21d9304c0aac25ce7dc6fdacf6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:34:08.534642 20632 raft_consensus.cc:3037] T f5c00121e5bb4e88b3a5e14392936713 P ca053d21d9304c0aac25ce7dc6fdacf6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:34:08.537124 20632 raft_consensus.cc:504] T f5c00121e5bb4e88b3a5e14392936713 P ca053d21d9304c0aac25ce7dc6fdacf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca053d21d9304c0aac25ce7dc6fdacf6" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45107 } } peers { permanent_uuid: "fa9aad2f0e384189872e794e35bbad29" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 34095 } } peers { permanent_uuid: "c6ed5f76dd9e4a4593e96a1eaad01553" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 33857 } }
I20260926 16:34:08.537899 20632 leader_election.cc:288] T f5c00121e5bb4e88b3a5e14392936713 P ca053d21d9304c0aac25ce7dc6fdacf6 [CANDIDATE]: Term 1 election: Requested vote from peers fa9aad2f0e384189872e794e35bbad29 (127.6.33.1:34095), c6ed5f76dd9e4a4593e96a1eaad01553 (127.6.33.2:33857)
I20260926 16:34:08.538383 20295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c00121e5bb4e88b3a5e14392936713" candidate_uuid: "ca053d21d9304c0aac25ce7dc6fdacf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa9aad2f0e384189872e794e35bbad29"
I20260926 16:34:08.538489 20429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c00121e5bb4e88b3a5e14392936713" candidate_uuid: "ca053d21d9304c0aac25ce7dc6fdacf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ed5f76dd9e4a4593e96a1eaad01553"
I20260926 16:34:08.538646 20295 raft_consensus.cc:3037] T f5c00121e5bb4e88b3a5e14392936713 P fa9aad2f0e384189872e794e35bbad29 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:34:08.538710 20429 raft_consensus.cc:3037] T f5c00121e5bb4e88b3a5e14392936713 P c6ed5f76dd9e4a4593e96a1eaad01553 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:34:08.542132 20429 raft_consensus.cc:2454] T f5c00121e5bb4e88b3a5e14392936713 P c6ed5f76dd9e4a4593e96a1eaad01553 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca053d21d9304c0aac25ce7dc6fdacf6 in term 1.
I20260926 16:34:08.542126 20295 raft_consensus.cc:2454] T f5c00121e5bb4e88b3a5e14392936713 P fa9aad2f0e384189872e794e35bbad29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca053d21d9304c0aac25ce7dc6fdacf6 in term 1.
I20260926 16:34:08.542687 20496 leader_election.cc:302] T f5c00121e5bb4e88b3a5e14392936713 P ca053d21d9304c0aac25ce7dc6fdacf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6ed5f76dd9e4a4593e96a1eaad01553, ca053d21d9304c0aac25ce7dc6fdacf6; no voters:
I20260926 16:34:08.543075 20632 raft_consensus.cc:2781] T f5c00121e5bb4e88b3a5e14392936713 P ca053d21d9304c0aac25ce7dc6fdacf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:34:08.543670 20632 raft_consensus.cc:686] T f5c00121e5bb4e88b3a5e14392936713 P ca053d21d9304c0aac25ce7dc6fdacf6 [term 1 LEADER]: Becoming Leader. State: Replica: ca053d21d9304c0aac25ce7dc6fdacf6, State: Running, Role: LEADER
I20260926 16:34:08.544296 20632 consensus_queue.cc:237] T f5c00121e5bb4e88b3a5e14392936713 P ca053d21d9304c0aac25ce7dc6fdacf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca053d21d9304c0aac25ce7dc6fdacf6" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45107 } } peers { permanent_uuid: "fa9aad2f0e384189872e794e35bbad29" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 34095 } } peers { permanent_uuid: "c6ed5f76dd9e4a4593e96a1eaad01553" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 33857 } }
I20260926 16:34:08.549360 20150 catalog_manager.cc:5495] T f5c00121e5bb4e88b3a5e14392936713 P ca053d21d9304c0aac25ce7dc6fdacf6 reported cstate change: term changed from 0 to 1, leader changed from <none> to ca053d21d9304c0aac25ce7dc6fdacf6 (127.6.33.3). New cstate: current_term: 1 leader_uuid: "ca053d21d9304c0aac25ce7dc6fdacf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca053d21d9304c0aac25ce7dc6fdacf6" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa9aad2f0e384189872e794e35bbad29" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 34095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6ed5f76dd9e4a4593e96a1eaad01553" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 33857 } health_report { overall_health: UNKNOWN } } }
W20260926 16:34:08.580861 20341 tablet.cc:2369] T f5c00121e5bb4e88b3a5e14392936713 P fa9aad2f0e384189872e794e35bbad29: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:34:08.594913 20621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:08.589316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44678 (local address 127.6.33.3:45107)
0926 16:34:08.589515 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:08.589519 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:08.589537 (+ 18us) server_negotiation.cc:407] Connection header received
0926 16:34:08.589633 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:08.589636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:08.589731 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:08.589857 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:34:08.590179 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:08.590985 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:08.592441 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:08.593054 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:08.593158 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:08.593656 (+ 498us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:34:08.594234 (+ 578us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:34:08.594354 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:08.594502 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 16:34:08.594588 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260926 16:34:08.601836 20475 tablet.cc:2369] T f5c00121e5bb4e88b3a5e14392936713 P c6ed5f76dd9e4a4593e96a1eaad01553: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:34:08.603645 20429 raft_consensus.cc:1261] T f5c00121e5bb4e88b3a5e14392936713 P c6ed5f76dd9e4a4593e96a1eaad01553 [term 1 FOLLOWER]: Refusing update from remote peer ca053d21d9304c0aac25ce7dc6fdacf6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:34:08.603717 20295 raft_consensus.cc:1261] T f5c00121e5bb4e88b3a5e14392936713 P fa9aad2f0e384189872e794e35bbad29 [term 1 FOLLOWER]: Refusing update from remote peer ca053d21d9304c0aac25ce7dc6fdacf6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:34:08.604703 20636 consensus_queue.cc:1035] T f5c00121e5bb4e88b3a5e14392936713 P ca053d21d9304c0aac25ce7dc6fdacf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ed5f76dd9e4a4593e96a1eaad01553" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 33857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:34:08.605193 20632 consensus_queue.cc:1035] T f5c00121e5bb4e88b3a5e14392936713 P ca053d21d9304c0aac25ce7dc6fdacf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa9aad2f0e384189872e794e35bbad29" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 16:34:08.605882 20609 tablet.cc:2369] T f5c00121e5bb4e88b3a5e14392936713 P ca053d21d9304c0aac25ce7dc6fdacf6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:34:08.632535 20151 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57846:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:34:08.632879 20151 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57846:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:34:08.638206 20151 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1681ae7327bb4f84a8620eb060da18ce: Sending DeleteTablet for 3 replicas of tablet f5c00121e5bb4e88b3a5e14392936713
I20260926 16:34:08.639473 20543 tablet_service.cc:1514] Processing DeleteTablet for tablet f5c00121e5bb4e88b3a5e14392936713 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:34:08 UTC) from {username='kudu', principal='kudu/127.6.33.62@KRBTEST.COM'} at 127.0.0.1:44676
I20260926 16:34:08.639487 20275 tablet_service.cc:1514] Processing DeleteTablet for tablet f5c00121e5bb4e88b3a5e14392936713 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:34:08 UTC) from {username='kudu', principal='kudu/127.6.33.62@KRBTEST.COM'} at 127.0.0.1:53546
I20260926 16:34:08.639581 20409 tablet_service.cc:1514] Processing DeleteTablet for tablet f5c00121e5bb4e88b3a5e14392936713 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:34:08 UTC) from {username='kudu', principal='kudu/127.6.33.62@KRBTEST.COM'} at 127.0.0.1:60898
I20260926 16:34:08.640882 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 20212
I20260926 16:34:08.645304 20650 tablet_replica.cc:330] T f5c00121e5bb4e88b3a5e14392936713 P ca053d21d9304c0aac25ce7dc6fdacf6: stopping tablet replica
I20260926 16:34:08.645844 20650 raft_consensus.cc:2229] T f5c00121e5bb4e88b3a5e14392936713 P ca053d21d9304c0aac25ce7dc6fdacf6 [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:34:08.646715 20650 raft_consensus.cc:2258] T f5c00121e5bb4e88b3a5e14392936713 P ca053d21d9304c0aac25ce7dc6fdacf6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:34:08.648995 20650 ts_tablet_manager.cc:1905] T f5c00121e5bb4e88b3a5e14392936713 P ca053d21d9304c0aac25ce7dc6fdacf6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:34:08.656991 20650 ts_tablet_manager.cc:1918] T f5c00121e5bb4e88b3a5e14392936713 P ca053d21d9304c0aac25ce7dc6fdacf6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:34:08.657331 20650 log.cc:1196] T f5c00121e5bb4e88b3a5e14392936713 P ca053d21d9304c0aac25ce7dc6fdacf6: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-2/wal/wals/f5c00121e5bb4e88b3a5e14392936713
I20260926 16:34:08.658031 20650 ts_tablet_manager.cc:1939] T f5c00121e5bb4e88b3a5e14392936713 P ca053d21d9304c0aac25ce7dc6fdacf6: Deleting consensus metadata
I20260926 16:34:08.660113 20137 catalog_manager.cc:4841] TS ca053d21d9304c0aac25ce7dc6fdacf6 (127.6.33.3:45107): tablet f5c00121e5bb4e88b3a5e14392936713 (table test-table [id=b8c6bd2f0d7948b38607bca8f4ba39cf]) successfully deleted
I20260926 16:34:08.667953 20649 tablet_replica.cc:330] T f5c00121e5bb4e88b3a5e14392936713 P fa9aad2f0e384189872e794e35bbad29: stopping tablet replica
I20260926 16:34:08.668625 20649 raft_consensus.cc:2229] T f5c00121e5bb4e88b3a5e14392936713 P fa9aad2f0e384189872e794e35bbad29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:34:08.669308 20649 raft_consensus.cc:2258] T f5c00121e5bb4e88b3a5e14392936713 P fa9aad2f0e384189872e794e35bbad29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:34:08.671866 20648 tablet_replica.cc:330] T f5c00121e5bb4e88b3a5e14392936713 P c6ed5f76dd9e4a4593e96a1eaad01553: stopping tablet replica
I20260926 16:34:08.672022 20649 ts_tablet_manager.cc:1905] T f5c00121e5bb4e88b3a5e14392936713 P fa9aad2f0e384189872e794e35bbad29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:34:08.672333 20648 raft_consensus.cc:2229] T f5c00121e5bb4e88b3a5e14392936713 P c6ed5f76dd9e4a4593e96a1eaad01553 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:34:08.672911 20648 raft_consensus.cc:2258] T f5c00121e5bb4e88b3a5e14392936713 P c6ed5f76dd9e4a4593e96a1eaad01553 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:34:08.675323 20648 ts_tablet_manager.cc:1905] T f5c00121e5bb4e88b3a5e14392936713 P c6ed5f76dd9e4a4593e96a1eaad01553: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 16:34:08 dist-test-slave-khd3 krb5kdc[20106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.1@KRBTEST.COM
I20260926 16:34:08.679277 20649 ts_tablet_manager.cc:1918] T f5c00121e5bb4e88b3a5e14392936713 P fa9aad2f0e384189872e794e35bbad29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:34:08.679562 20649 log.cc:1196] T f5c00121e5bb4e88b3a5e14392936713 P fa9aad2f0e384189872e794e35bbad29: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-0/wal/wals/f5c00121e5bb4e88b3a5e14392936713
I20260926 16:34:08.680179 20649 ts_tablet_manager.cc:1939] T f5c00121e5bb4e88b3a5e14392936713 P fa9aad2f0e384189872e794e35bbad29: Deleting consensus metadata
I20260926 16:34:08.682191 20138 catalog_manager.cc:4841] TS fa9aad2f0e384189872e794e35bbad29 (127.6.33.1:34095): tablet f5c00121e5bb4e88b3a5e14392936713 (table test-table [id=b8c6bd2f0d7948b38607bca8f4ba39cf]) successfully deleted
I20260926 16:34:08.683449 20648 ts_tablet_manager.cc:1918] T f5c00121e5bb4e88b3a5e14392936713 P c6ed5f76dd9e4a4593e96a1eaad01553: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:34:08.683646 20648 log.cc:1196] T f5c00121e5bb4e88b3a5e14392936713 P c6ed5f76dd9e4a4593e96a1eaad01553: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-1/wal/wals/f5c00121e5bb4e88b3a5e14392936713
I20260926 16:34:08.684217 20648 ts_tablet_manager.cc:1939] T f5c00121e5bb4e88b3a5e14392936713 P c6ed5f76dd9e4a4593e96a1eaad01553: Deleting consensus metadata
I20260926 16:34:08.685858 20623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:08.641847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53552 (local address 127.6.33.1:34095)
0926 16:34:08.642053 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:08.642063 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:08.661415 (+ 19352us) server_negotiation.cc:407] Connection header received
0926 16:34:08.661724 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:08.661730 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:08.661852 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:08.661932 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:34:08.664600 (+ 2668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:08.665793 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:08.669045 (+ 3252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:08.669998 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:08.670147 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:08.677784 (+ 7637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:34:08.677799 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:34:08.677818 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:34:08.677875 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:34:08.682021 (+ 4146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:08.683051 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:08.683064 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:08.683067 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:08.683257 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:08.683784 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:08.683790 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:08.683795 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:08.684298 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:34:08.684591 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:08.685242 (+ 651us) server_negotiation.cc:299] Negotiation successful
0926 16:34:08.685488 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 16:34:08.686196 20135 catalog_manager.cc:4841] TS c6ed5f76dd9e4a4593e96a1eaad01553 (127.6.33.2:33857): tablet f5c00121e5bb4e88b3a5e14392936713 (table test-table [id=b8c6bd2f0d7948b38607bca8f4ba39cf]) successfully deleted
I20260926 16:34:08.687366 20335 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:34:08.779332 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 20212
I20260926 16:34:08.797200 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 20346
Sep 26 16:34:08 dist-test-slave-khd3 krb5kdc[20106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.2@KRBTEST.COM
I20260926 16:34:08.815016 20624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:08.798098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60910 (local address 127.6.33.2:33857)
0926 16:34:08.798306 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:08.798313 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:08.798336 (+ 23us) server_negotiation.cc:407] Connection header received
0926 16:34:08.798430 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:08.798436 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:08.798537 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:08.798645 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:34:08.799840 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:08.801147 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:08.802908 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:08.803549 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:08.803704 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:08.808150 (+ 4446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:34:08.808170 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:34:08.808193 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:34:08.808245 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:34:08.812020 (+ 3775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:08.812739 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:08.812749 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:08.812753 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:08.812933 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:08.813299 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:08.813304 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:08.813308 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:08.813818 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:34:08.814099 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:08.814369 (+ 270us) server_negotiation.cc:299] Negotiation successful
0926 16:34:08.814612 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 16:34:08.815789 20469 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:34:08.907881 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 20346
I20260926 16:34:08.931856 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 20480
Sep 26 16:34:08 dist-test-slave-khd3 krb5kdc[20106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.3@KRBTEST.COM
I20260926 16:34:08.951336 20621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:08.932690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44690 (local address 127.6.33.3:45107)
0926 16:34:08.932876 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:08.932881 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:08.932900 (+ 19us) server_negotiation.cc:407] Connection header received
0926 16:34:08.933139 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:08.933143 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:08.933212 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:08.933303 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:34:08.935178 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:08.935997 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:08.938199 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:08.938804 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:08.939022 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:08.944619 (+ 5597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:34:08.944632 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:34:08.944650 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:34:08.944691 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:34:08.948293 (+ 3602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:08.949229 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:08.949237 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:08.949239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:08.949402 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:08.949820 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:08.949823 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:08.949825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:08.950182 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:34:08.950389 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:08.950764 (+ 375us) server_negotiation.cc:299] Negotiation successful
0926 16:34:08.950928 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 16:34:08.952046 20603 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:34:09.075330 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 20480
I20260926 16:34:09.094888 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 20119
I20260926 16:34:09.096361 20181 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:34:09.208575 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 20119
I20260926 16:34:09.231050 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:36283
--webserver_interface=127.6.33.62
--webserver_port=46143
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:44077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.33.62:36283
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:34:09.570034 20656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:34:09.570345 20656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:34:09.570436 20656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:34:09.578099 20656 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:34:09.578235 20656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:34:09.578284 20656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:34:09.578334 20656 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:34:09.578379 20656 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:34:09.588914 20656 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:44077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.6.33.62:36283
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:36283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.33.62
--webserver_port=46143
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:34:09.590229 20656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:34:09.591530 20656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:34:09.614705 20662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:34:09.614755 20665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:34:09.614733 20663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:34:09.615696 20656 server_base.cc:1034] running on GCE node
I20260926 16:34:09.616588 20656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:34:09.618350 20656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:34:09.619593 20656 hybrid_clock.cc:648] HybridClock initialized: now 1790440449619540 us; error 74 us; skew 500 ppm
I20260926 16:34:09.620015 20656 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:34:09 dist-test-slave-khd3 krb5kdc[20106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440449, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:34:09.624492 20656 init.cc:377] Logged in from keytab as kudu/127.6.33.62@KRBTEST.COM (short username kudu)
I20260926 16:34:09.626946 20656 webserver.cc:460] Webserver started at http://127.6.33.62:46143/ using document root <none> and password file <none>
I20260926 16:34:09.627974 20656 fs_manager.cc:359] Metadata directory not provided
I20260926 16:34:09.628139 20656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:34:09.635401 20656 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260926 16:34:09.640085 20672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:34:09.641852 20656 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260926 16:34:09.642066 20656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/master-0/wal
uuid: "1681ae7327bb4f84a8620eb060da18ce"
format_stamp: "Formatted at 2026-09-26 16:34:06 on dist-test-slave-khd3"
server_key: "31d371d57d2a98fc3e204258b17c14d6"
server_key_iv: "b7c4e0b0d51311bcbb459f9afb5d10a9"
server_key_version: "encryptionkey@0"
I20260926 16:34:09.643265 20656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:34:09.666347 20656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:34:09.671063 20656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:34:09.671397 20656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:34:09.691640 20656 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.62:36283
I20260926 16:34:09.691658 20723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.62:36283 every 8 connection(s)
I20260926 16:34:09.693390 20656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
I20260926 16:34:09.694931 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 20656
I20260926 16:34:09.695735 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.1:34095
--local_ip_for_outbound_sockets=127.6.33.1
--tserver_master_addrs=127.6.33.62:36283
--webserver_port=41611
--webserver_interface=127.6.33.1
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab
--principal=kudu/127.6.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:44077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:34:09.701565 20724 sys_catalog.cc:263] Verifying existing consensus state
I20260926 16:34:09.707502 20724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1681ae7327bb4f84a8620eb060da18ce: Bootstrap starting.
I20260926 16:34:09.740651 20724 log.cc:824] T 00000000000000000000000000000000 P 1681ae7327bb4f84a8620eb060da18ce: Log is configured to *not* fsync() on all Append() calls
I20260926 16:34:09.757742 20724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1681ae7327bb4f84a8620eb060da18ce: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 16:34:09.758538 20724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1681ae7327bb4f84a8620eb060da18ce: Bootstrap complete.
I20260926 16:34:09.774195 20724 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1681ae7327bb4f84a8620eb060da18ce [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1681ae7327bb4f84a8620eb060da18ce" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 36283 } }
I20260926 16:34:09.775442 20724 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1681ae7327bb4f84a8620eb060da18ce [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1681ae7327bb4f84a8620eb060da18ce, State: Initialized, Role: FOLLOWER
I20260926 16:34:09.776224 20724 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1681ae7327bb4f84a8620eb060da18ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1681ae7327bb4f84a8620eb060da18ce" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 36283 } }
I20260926 16:34:09.776542 20724 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1681ae7327bb4f84a8620eb060da18ce [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:34:09.776674 20724 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1681ae7327bb4f84a8620eb060da18ce [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:34:09.776897 20724 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1681ae7327bb4f84a8620eb060da18ce [term 1 FOLLOWER]: Advancing to term 2
I20260926 16:34:09.783008 20724 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1681ae7327bb4f84a8620eb060da18ce [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1681ae7327bb4f84a8620eb060da18ce" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 36283 } }
I20260926 16:34:09.783744 20724 leader_election.cc:302] T 00000000000000000000000000000000 P 1681ae7327bb4f84a8620eb060da18ce [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1681ae7327bb4f84a8620eb060da18ce; no voters:
I20260926 16:34:09.784444 20724 leader_election.cc:288] T 00000000000000000000000000000000 P 1681ae7327bb4f84a8620eb060da18ce [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 16:34:09.784641 20728 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1681ae7327bb4f84a8620eb060da18ce [term 2 FOLLOWER]: Leader election won for term 2
I20260926 16:34:09.785864 20728 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1681ae7327bb4f84a8620eb060da18ce [term 2 LEADER]: Becoming Leader. State: Replica: 1681ae7327bb4f84a8620eb060da18ce, State: Running, Role: LEADER
I20260926 16:34:09.786617 20728 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1681ae7327bb4f84a8620eb060da18ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1681ae7327bb4f84a8620eb060da18ce" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 36283 } }
I20260926 16:34:09.786846 20724 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1681ae7327bb4f84a8620eb060da18ce [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:34:09.789866 20729 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1681ae7327bb4f84a8620eb060da18ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1681ae7327bb4f84a8620eb060da18ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1681ae7327bb4f84a8620eb060da18ce" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 36283 } } }
I20260926 16:34:09.790447 20729 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1681ae7327bb4f84a8620eb060da18ce [sys.catalog]: This master's current role is: LEADER
I20260926 16:34:09.791219 20730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1681ae7327bb4f84a8620eb060da18ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1681ae7327bb4f84a8620eb060da18ce. Latest consensus state: current_term: 2 leader_uuid: "1681ae7327bb4f84a8620eb060da18ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1681ae7327bb4f84a8620eb060da18ce" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 36283 } } }
I20260926 16:34:09.791592 20730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1681ae7327bb4f84a8620eb060da18ce [sys.catalog]: This master's current role is: LEADER
I20260926 16:34:09.799300 20735 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:34:09.810465 20735 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:34:09.812656 20735 catalog_manager.cc:1259] Loaded cluster ID: 4a47e9ca2a544b0aa9e2c015c3a18ef4
I20260926 16:34:09.813265 20735 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:34:09.818413 20735 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260926 16:34:09.834926 20735 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260926 16:34:09.842422 20735 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:34:09.844092 20735 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1681ae7327bb4f84a8620eb060da18ce: Loaded TSK: 0
I20260926 16:34:09.845345 20735 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 16:34:10.104507 20726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:34:10.104911 20726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:34:10.105093 20726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:34:10.113322 20726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:34:10.113479 20726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:34:10.113660 20726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.1
I20260926 16:34:10.124457 20726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:44077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab
--principal=kudu/127.6.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.1:34095
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.33.1
--webserver_port=41611
--tserver_master_addrs=127.6.33.62:36283
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.1
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:34:10.125947 20726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:34:10.127349 20726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:34:10.142799 20752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:34:10.142899 20751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:34:10.143415 20754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:34:10.143966 20726 server_base.cc:1034] running on GCE node
I20260926 16:34:10.144829 20726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:34:10.146179 20726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:34:10.147439 20726 hybrid_clock.cc:648] HybridClock initialized: now 1790440450147360 us; error 67 us; skew 500 ppm
I20260926 16:34:10.147848 20726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:34:10 dist-test-slave-khd3 krb5kdc[20106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440450, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:34:10.158965 20726 init.cc:377] Logged in from keytab as kudu/127.6.33.1@KRBTEST.COM (short username kudu)
I20260926 16:34:10.160863 20726 webserver.cc:460] Webserver started at http://127.6.33.1:41611/ using document root <none> and password file <none>
I20260926 16:34:10.161665 20726 fs_manager.cc:359] Metadata directory not provided
I20260926 16:34:10.161820 20726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:34:10.168085 20726 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 16:34:10.171267 20761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:34:10.172859 20726 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:34:10.173058 20726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-0/wal
uuid: "fa9aad2f0e384189872e794e35bbad29"
format_stamp: "Formatted at 2026-09-26 16:34:07 on dist-test-slave-khd3"
server_key: "0a324ffecaed301b7df470ea749a8246"
server_key_iv: "9e692dee5188eb253425075ecc5d2f6c"
server_key_version: "encryptionkey@0"
I20260926 16:34:10.174153 20726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:34:10.202370 20726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:34:10.207465 20726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:34:10.207787 20726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:34:10.208976 20726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:34:10.210393 20726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:34:10.210501 20726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:34:10.210582 20726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:34:10.210637 20726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:34:10.245775 20726 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.1:34095
I20260926 16:34:10.245986 20873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.1:34095 every 8 connection(s)
I20260926 16:34:10.247465 20726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
I20260926 16:34:10.255645 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 20726
I20260926 16:34:10.256394 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.2:33857
--local_ip_for_outbound_sockets=127.6.33.2
--tserver_master_addrs=127.6.33.62:36283
--webserver_port=38415
--webserver_interface=127.6.33.2
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab
--principal=kudu/127.6.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:44077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 16:34:10 dist-test-slave-khd3 krb5kdc[20106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440450, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.1@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:34:10.273388 20876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:10.250576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:36151 (local address 127.6.33.62:36283)
0926 16:34:10.251194 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:10.251207 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:10.251688 (+ 481us) server_negotiation.cc:407] Connection header received
0926 16:34:10.253902 (+ 2214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:10.253916 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:10.254291 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:10.254604 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:34:10.256273 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:10.258462 (+ 2189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:10.260015 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:10.261219 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:10.261440 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:10.265197 (+ 3757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:34:10.265206 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:34:10.265223 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:34:10.265269 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:34:10.268389 (+ 3120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:10.269363 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:10.269370 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:10.269376 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:10.269555 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:10.270043 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:10.270046 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:10.270048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:10.270556 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:34:10.270777 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:10.271588 (+ 811us) server_negotiation.cc:299] Negotiation successful
0926 16:34:10.271993 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":277,"threads_started":1}
I20260926 16:34:10.273388 20877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:10.250677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36283 (local address 127.6.33.1:36151)
0926 16:34:10.251491 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:10.251540 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 16:34:10.253358 (+ 1818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:34:10.254791 (+ 1433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:34:10.254799 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:34:10.255373 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:34:10.256020 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:10.256038 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:10.258624 (+ 2586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:10.258628 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:10.259861 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:10.259873 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:10.261432 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:10.261438 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:10.261650 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:10.262384 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:34:10.262415 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:34:10.264889 (+ 2474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:34:10.268579 (+ 3690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:34:10.268589 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:34:10.268607 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:34:10.269192 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:34:10.269687 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:34:10.269693 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:34:10.269697 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:34:10.269905 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:34:10.270713 (+ 808us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:34:10.270721 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:34:10.271080 (+ 359us) client_negotiation.cc:769] Sending connection context
0926 16:34:10.271357 (+ 277us) client_negotiation.cc:240] Negotiation successful
0926 16:34:10.271965 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":344,"threads_started":1}
I20260926 16:34:10.276659 20874 heartbeater.cc:347] Connected to a master server at 127.6.33.62:36283
I20260926 16:34:10.277109 20874 heartbeater.cc:464] Registering TS with master...
I20260926 16:34:10.278249 20874 heartbeater.cc:511] Master 127.6.33.62:36283 requested a full tablet report, sending...
I20260926 16:34:10.280895 20689 ts_manager.cc:194] Registered new tserver with Master: fa9aad2f0e384189872e794e35bbad29 (127.6.33.1:34095)
I20260926 16:34:10.282462 20689 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.33.1@KRBTEST.COM'} at 127.6.33.1:36151
W20260926 16:34:10.601364 20878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:34:10.601691 20878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:34:10.601791 20878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:34:10.609586 20878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:34:10.609709 20878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:34:10.609889 20878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.2
I20260926 16:34:10.620316 20878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:44077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab
--principal=kudu/127.6.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.2:33857
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.33.2
--webserver_port=38415
--tserver_master_addrs=127.6.33.62:36283
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.2
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:34:10.621788 20878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:34:10.623245 20878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:34:10.636631 20886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:34:10.636749 20883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:34:10.636770 20884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:34:10.637655 20878 server_base.cc:1034] running on GCE node
I20260926 16:34:10.638285 20878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:34:10.639405 20878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:34:10.640656 20878 hybrid_clock.cc:648] HybridClock initialized: now 1790440450640595 us; error 46 us; skew 500 ppm
I20260926 16:34:10.641038 20878 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:34:10 dist-test-slave-khd3 krb5kdc[20106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440450, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:34:10.645430 20878 init.cc:377] Logged in from keytab as kudu/127.6.33.2@KRBTEST.COM (short username kudu)
I20260926 16:34:10.647199 20878 webserver.cc:460] Webserver started at http://127.6.33.2:38415/ using document root <none> and password file <none>
I20260926 16:34:10.648068 20878 fs_manager.cc:359] Metadata directory not provided
I20260926 16:34:10.648247 20878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:34:10.654006 20878 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 16:34:10.657015 20893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:34:10.658532 20878 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 16:34:10.658735 20878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-1/wal
uuid: "c6ed5f76dd9e4a4593e96a1eaad01553"
format_stamp: "Formatted at 2026-09-26 16:34:07 on dist-test-slave-khd3"
server_key: "cd5beecea99a70c7b4a489beb613869e"
server_key_iv: "5a67ea834ccffb23601ab2c028fdf783"
server_key_version: "encryptionkey@0"
I20260926 16:34:10.659890 20878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:34:10.690351 20878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:34:10.695255 20878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:34:10.695580 20878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:34:10.696609 20878 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:34:10.698186 20878 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:34:10.698311 20878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:34:10.698493 20878 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:34:10.698587 20878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:34:10.732664 20878 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.2:33857
I20260926 16:34:10.732798 21005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.2:33857 every 8 connection(s)
I20260926 16:34:10.734197 20878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
I20260926 16:34:10.742172 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 20878
I20260926 16:34:10.742897 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.3:45107
--local_ip_for_outbound_sockets=127.6.33.3
--tserver_master_addrs=127.6.33.62:36283
--webserver_port=36299
--webserver_interface=127.6.33.3
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab
--principal=kudu/127.6.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:44077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 16:34:10 dist-test-slave-khd3 krb5kdc[20106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440450, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.2@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:34:10.755321 20876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:10.736919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:58291 (local address 127.6.33.62:36283)
0926 16:34:10.737095 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:10.737099 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:10.738329 (+ 1230us) server_negotiation.cc:407] Connection header received
0926 16:34:10.740147 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:10.740152 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:10.740268 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:10.740365 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:34:10.741961 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:10.742718 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:10.744393 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:10.744890 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:10.745029 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:10.748728 (+ 3699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:34:10.748733 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:34:10.748736 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:34:10.748818 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:34:10.751602 (+ 2784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:10.752621 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:10.752626 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:10.752628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:10.752830 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:10.753397 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:10.753401 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:10.753404 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:10.753698 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:34:10.753822 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:10.754788 (+ 966us) server_negotiation.cc:299] Negotiation successful
0926 16:34:10.755054 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 16:34:10.756932 21008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:10.737435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36283 (local address 127.6.33.2:58291)
0926 16:34:10.738158 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:10.738196 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 16:34:10.739855 (+ 1659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:34:10.740570 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:34:10.740578 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:34:10.741201 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:34:10.741771 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:10.741789 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:10.742882 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:10.742887 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:10.744226 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:10.744249 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:10.745122 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:10.745130 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:10.745369 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:10.746078 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:34:10.746100 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:34:10.748493 (+ 2393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:34:10.751774 (+ 3281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:34:10.751787 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:34:10.751809 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:34:10.752480 (+ 671us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:34:10.752946 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:34:10.752954 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:34:10.752961 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:34:10.753229 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:34:10.753843 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:34:10.753854 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:34:10.754342 (+ 488us) client_negotiation.cc:769] Sending connection context
0926 16:34:10.754677 (+ 335us) client_negotiation.cc:240] Negotiation successful
0926 16:34:10.755303 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":294,"threads_started":1}
I20260926 16:34:10.758661 21006 heartbeater.cc:347] Connected to a master server at 127.6.33.62:36283
I20260926 16:34:10.759021 21006 heartbeater.cc:464] Registering TS with master...
I20260926 16:34:10.760003 21006 heartbeater.cc:511] Master 127.6.33.62:36283 requested a full tablet report, sending...
I20260926 16:34:10.762027 20689 ts_manager.cc:194] Registered new tserver with Master: c6ed5f76dd9e4a4593e96a1eaad01553 (127.6.33.2:33857)
I20260926 16:34:10.762884 20689 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.33.2@KRBTEST.COM'} at 127.6.33.2:58291
W20260926 16:34:11.091939 21009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:34:11.092303 21009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:34:11.092522 21009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:34:11.100157 21009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:34:11.100315 21009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:34:11.100507 21009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.3
I20260926 16:34:11.110471 21009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:44077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab
--principal=kudu/127.6.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.3:45107
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.33.3
--webserver_port=36299
--tserver_master_addrs=127.6.33.62:36283
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.3
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:34:11.111841 21009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:34:11.113658 21009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:34:11.126855 21014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:34:11.126885 21017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:34:11.126847 21015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:34:11.127667 21009 server_base.cc:1034] running on GCE node
I20260926 16:34:11.128324 21009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:34:11.129333 21009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:34:11.130566 21009 hybrid_clock.cc:648] HybridClock initialized: now 1790440451130500 us; error 66 us; skew 500 ppm
I20260926 16:34:11.130878 21009 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:34:11 dist-test-slave-khd3 krb5kdc[20106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440451, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:34:11.134343 21009 init.cc:377] Logged in from keytab as kudu/127.6.33.3@KRBTEST.COM (short username kudu)
I20260926 16:34:11.135911 21009 webserver.cc:460] Webserver started at http://127.6.33.3:36299/ using document root <none> and password file <none>
I20260926 16:34:11.136656 21009 fs_manager.cc:359] Metadata directory not provided
I20260926 16:34:11.136833 21009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:34:11.142359 21009 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 16:34:11.145278 21024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:34:11.147126 21009 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 16:34:11.147430 21009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-2/wal
uuid: "ca053d21d9304c0aac25ce7dc6fdacf6"
format_stamp: "Formatted at 2026-09-26 16:34:08 on dist-test-slave-khd3"
server_key: "95608f2856ca05c3a623b28ff261dea6"
server_key_iv: "3cebd328b8ea9a153efe2e27eccbd748"
server_key_version: "encryptionkey@0"
I20260926 16:34:11.148494 21009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:34:11.161815 21009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:34:11.167258 21009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:34:11.167618 21009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:34:11.168699 21009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:34:11.170241 21009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:34:11.170337 21009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:34:11.170490 21009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:34:11.170584 21009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:34:11.202674 21009 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.3:45107
I20260926 16:34:11.202764 21136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.3:45107 every 8 connection(s)
I20260926 16:34:11.204182 21009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
I20260926 16:34:11.208981 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 21009
Sep 26 16:34:11 dist-test-slave-khd3 krb5kdc[20106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440451, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.3@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:34:11.229687 20876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:11.207133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.3:34059 (local address 127.6.33.62:36283)
0926 16:34:11.207363 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:11.207368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:11.208654 (+ 1286us) server_negotiation.cc:407] Connection header received
0926 16:34:11.211075 (+ 2421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:11.211084 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:11.211187 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:11.211327 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:34:11.213115 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:11.214652 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:11.216299 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:11.217258 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:11.217428 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:11.222655 (+ 5227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:34:11.222660 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:34:11.222664 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:34:11.222699 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:34:11.225839 (+ 3140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:11.226908 (+ 1069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:11.226912 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:11.226914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:11.227036 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:11.227570 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:11.227573 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:11.227575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:11.228002 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:34:11.228167 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:11.229024 (+ 857us) server_negotiation.cc:299] Negotiation successful
0926 16:34:11.229300 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260926 16:34:11.230732 21139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:11.207663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36283 (local address 127.6.33.3:34059)
0926 16:34:11.208462 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:11.208507 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 16:34:11.210656 (+ 2149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:34:11.211635 (+ 979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:34:11.211645 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:34:11.212216 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:34:11.212916 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:11.212940 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:11.214842 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:11.214846 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:11.216138 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:11.216151 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:11.217527 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:11.217533 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:11.217761 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:11.219169 (+ 1408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:34:11.219213 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:34:11.222405 (+ 3192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:34:11.226019 (+ 3614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:34:11.226034 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:34:11.226054 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:34:11.226767 (+ 713us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:34:11.227161 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:34:11.227167 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:34:11.227171 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:34:11.227436 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:34:11.228163 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:34:11.228170 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:34:11.228531 (+ 361us) client_negotiation.cc:769] Sending connection context
0926 16:34:11.228892 (+ 361us) client_negotiation.cc:240] Negotiation successful
0926 16:34:11.229494 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":327,"threads_started":1}
I20260926 16:34:11.232439 21137 heartbeater.cc:347] Connected to a master server at 127.6.33.62:36283
I20260926 16:34:11.232837 21137 heartbeater.cc:464] Registering TS with master...
I20260926 16:34:11.233611 21137 heartbeater.cc:511] Master 127.6.33.62:36283 requested a full tablet report, sending...
I20260926 16:34:11.234043 21141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:11.210141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57850 (local address 127.6.33.62:36283)
0926 16:34:11.218538 (+ 8397us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:11.218544 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:11.218564 (+ 20us) server_negotiation.cc:407] Connection header received
0926 16:34:11.218662 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:11.218669 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:11.218749 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:11.218866 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:34:11.221079 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:11.222101 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:11.223704 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:11.224451 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:11.224566 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:11.228011 (+ 3445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:34:11.228016 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:34:11.228020 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:34:11.228058 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:34:11.231051 (+ 2993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:11.231800 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:11.231807 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:11.231811 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:11.231964 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:11.232348 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:11.232354 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:11.232357 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:11.232676 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:34:11.232959 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:11.233270 (+ 311us) server_negotiation.cc:299] Negotiation successful
0926 16:34:11.233500 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8251,"thread_start_us":8092,"threads_started":1}
I20260926 16:34:11.235652 20689 ts_manager.cc:194] Registered new tserver with Master: ca053d21d9304c0aac25ce7dc6fdacf6 (127.6.33.3:45107)
I20260926 16:34:11.236963 6276 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:34:11.237004 20689 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.33.3@KRBTEST.COM'} at 127.6.33.3:34059
I20260926 16:34:11.243851 21141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:11.238059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57862 (local address 127.6.33.62:36283)
0926 16:34:11.238241 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:11.238249 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:11.238621 (+ 372us) server_negotiation.cc:407] Connection header received
0926 16:34:11.238870 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:11.238897 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:11.239026 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:11.239170 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:34:11.239607 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:11.240290 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:11.241794 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:11.242221 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:11.242347 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:11.242827 (+ 480us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:34:11.243302 (+ 475us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:34:11.243466 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:11.243538 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 16:34:11.243617 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 16:34:11.244850 20689 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:57862:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:34:11.254781 20689 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:34:11.279989 21150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:11.267157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44702 (local address 127.6.33.3:45107)
0926 16:34:11.270676 (+ 3519us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:11.270688 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:11.270733 (+ 45us) server_negotiation.cc:407] Connection header received
0926 16:34:11.270869 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:11.270882 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:11.271389 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:11.271612 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:34:11.272278 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:11.274087 (+ 1809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:11.276793 (+ 2706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:11.278505 (+ 1712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:11.278688 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:11.278961 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:11.279412 (+ 451us) server_negotiation.cc:299] Negotiation successful
0926 16:34:11.279623 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3295,"thread_start_us":3082,"threads_started":1}
I20260926 16:34:11.280002 21147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:11.266895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.3:45107 (local address 127.0.0.1:44702)
0926 16:34:11.268794 (+ 1899us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:11.268841 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 16:34:11.269036 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:34:11.271625 (+ 2589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:34:11.271641 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:34:11.271678 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:34:11.272067 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:11.272078 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:11.274259 (+ 2181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:11.274265 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:11.276580 (+ 2315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:11.276594 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:11.278734 (+ 2140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:11.278741 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:11.278960 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:11.279031 (+ 71us) client_negotiation.cc:769] Sending connection context
0926 16:34:11.279228 (+ 197us) client_negotiation.cc:240] Negotiation successful
0926 16:34:11.279478 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1558,"mutex_wait_us":171,"thread_start_us":1954,"threads_started":1}
I20260926 16:34:11.280951 21148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:11.266943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60926 (local address 127.6.33.2:33857)
0926 16:34:11.267637 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:11.267650 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:11.271052 (+ 3402us) server_negotiation.cc:407] Connection header received
0926 16:34:11.271414 (+ 362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:11.271422 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:11.271678 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:11.271842 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:34:11.272797 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:11.274614 (+ 1817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:11.277856 (+ 3242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:11.279523 (+ 1667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:11.279648 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:11.279822 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:11.280412 (+ 590us) server_negotiation.cc:299] Negotiation successful
0926 16:34:11.280595 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":395,"threads_started":1}
I20260926 16:34:11.281745 21151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:11.266895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.2:33857 (local address 127.0.0.1:60926)
0926 16:34:11.270934 (+ 4039us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:11.270954 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 16:34:11.271189 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:34:11.272139 (+ 950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:34:11.272143 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:34:11.272169 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:34:11.272565 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:11.272578 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:11.274766 (+ 2188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:11.274773 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:11.277678 (+ 2905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:11.277691 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:11.279989 (+ 2298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:11.279995 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:11.280208 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:11.280239 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 16:34:11.281165 (+ 926us) client_negotiation.cc:240] Negotiation successful
0926 16:34:11.281308 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3911,"mutex_wait_us":375,"thread_start_us":4140,"threads_started":1}
I20260926 16:34:11.282262 21152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:11.268653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53564 (local address 127.6.33.1:34095)
0926 16:34:11.269979 (+ 1326us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:11.269989 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:11.271614 (+ 1625us) server_negotiation.cc:407] Connection header received
0926 16:34:11.271930 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:11.271938 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:11.272142 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:11.272308 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:34:11.273195 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:11.274664 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:11.277711 (+ 3047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:11.279387 (+ 1676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:11.279517 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:11.279769 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:11.281693 (+ 1924us) server_negotiation.cc:299] Negotiation successful
0926 16:34:11.281904 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1136,"thread_start_us":1164,"threads_started":1}
I20260926 16:34:11.282647 21149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:11.268342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:34095 (local address 127.0.0.1:53564)
0926 16:34:11.271512 (+ 3170us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:11.271527 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 16:34:11.271729 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:34:11.272595 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:34:11.272603 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:34:11.272639 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:34:11.273008 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:11.273016 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:11.274842 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:11.274847 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:11.277511 (+ 2664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:11.277528 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:11.279947 (+ 2419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:11.279956 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:11.281448 (+ 1492us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:11.281485 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 16:34:11.281591 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 16:34:11.281744 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3059,"mutex_wait_us":176,"thread_start_us":2302,"threads_started":1}
I20260926 16:34:11.284901 21072 tablet_service.cc:1467] Processing CreateTablet for tablet e57c32124a154d2bbfccf97dcf62dcfa (DEFAULT_TABLE table=test-table [id=dfc45ca7622847be80f35776a8066351]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:34:11.286296 20874 heartbeater.cc:503] Master 127.6.33.62:36283 was elected leader, sending a full tablet report...
I20260926 16:34:11.286568 21072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e57c32124a154d2bbfccf97dcf62dcfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:34:11.286288 20809 tablet_service.cc:1467] Processing CreateTablet for tablet e57c32124a154d2bbfccf97dcf62dcfa (DEFAULT_TABLE table=test-table [id=dfc45ca7622847be80f35776a8066351]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:34:11.286621 20941 tablet_service.cc:1467] Processing CreateTablet for tablet e57c32124a154d2bbfccf97dcf62dcfa (DEFAULT_TABLE table=test-table [id=dfc45ca7622847be80f35776a8066351]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:34:11.288025 20809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e57c32124a154d2bbfccf97dcf62dcfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:34:11.288056 20941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e57c32124a154d2bbfccf97dcf62dcfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:34:11.300364 21153 tablet_bootstrap.cc:492] T e57c32124a154d2bbfccf97dcf62dcfa P fa9aad2f0e384189872e794e35bbad29: Bootstrap starting.
I20260926 16:34:11.304451 21155 tablet_bootstrap.cc:492] T e57c32124a154d2bbfccf97dcf62dcfa P c6ed5f76dd9e4a4593e96a1eaad01553: Bootstrap starting.
I20260926 16:34:11.304999 21153 tablet_bootstrap.cc:654] T e57c32124a154d2bbfccf97dcf62dcfa P fa9aad2f0e384189872e794e35bbad29: Neither blocks nor log segments found. Creating new log.
I20260926 16:34:11.306547 21154 tablet_bootstrap.cc:492] T e57c32124a154d2bbfccf97dcf62dcfa P ca053d21d9304c0aac25ce7dc6fdacf6: Bootstrap starting.
I20260926 16:34:11.306530 21153 log.cc:824] T e57c32124a154d2bbfccf97dcf62dcfa P fa9aad2f0e384189872e794e35bbad29: Log is configured to *not* fsync() on all Append() calls
I20260926 16:34:11.308244 21155 tablet_bootstrap.cc:654] T e57c32124a154d2bbfccf97dcf62dcfa P c6ed5f76dd9e4a4593e96a1eaad01553: Neither blocks nor log segments found. Creating new log.
I20260926 16:34:11.309659 21155 log.cc:824] T e57c32124a154d2bbfccf97dcf62dcfa P c6ed5f76dd9e4a4593e96a1eaad01553: Log is configured to *not* fsync() on all Append() calls
I20260926 16:34:11.310030 21154 tablet_bootstrap.cc:654] T e57c32124a154d2bbfccf97dcf62dcfa P ca053d21d9304c0aac25ce7dc6fdacf6: Neither blocks nor log segments found. Creating new log.
I20260926 16:34:11.310688 21153 tablet_bootstrap.cc:492] T e57c32124a154d2bbfccf97dcf62dcfa P fa9aad2f0e384189872e794e35bbad29: No bootstrap required, opened a new log
I20260926 16:34:11.311007 21153 ts_tablet_manager.cc:1397] T e57c32124a154d2bbfccf97dcf62dcfa P fa9aad2f0e384189872e794e35bbad29: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.003s
I20260926 16:34:11.311357 21154 log.cc:824] T e57c32124a154d2bbfccf97dcf62dcfa P ca053d21d9304c0aac25ce7dc6fdacf6: Log is configured to *not* fsync() on all Append() calls
I20260926 16:34:11.312906 21155 tablet_bootstrap.cc:492] T e57c32124a154d2bbfccf97dcf62dcfa P c6ed5f76dd9e4a4593e96a1eaad01553: No bootstrap required, opened a new log
I20260926 16:34:11.313192 21155 ts_tablet_manager.cc:1397] T e57c32124a154d2bbfccf97dcf62dcfa P c6ed5f76dd9e4a4593e96a1eaad01553: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260926 16:34:11.314606 21154 tablet_bootstrap.cc:492] T e57c32124a154d2bbfccf97dcf62dcfa P ca053d21d9304c0aac25ce7dc6fdacf6: No bootstrap required, opened a new log
I20260926 16:34:11.314869 21154 ts_tablet_manager.cc:1397] T e57c32124a154d2bbfccf97dcf62dcfa P ca053d21d9304c0aac25ce7dc6fdacf6: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 16:34:11.319378 21153 raft_consensus.cc:348] T e57c32124a154d2bbfccf97dcf62dcfa P fa9aad2f0e384189872e794e35bbad29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ed5f76dd9e4a4593e96a1eaad01553" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 33857 } } peers { permanent_uuid: "ca053d21d9304c0aac25ce7dc6fdacf6" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45107 } } peers { permanent_uuid: "fa9aad2f0e384189872e794e35bbad29" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 34095 } }
I20260926 16:34:11.319789 21153 raft_consensus.cc:374] T e57c32124a154d2bbfccf97dcf62dcfa P fa9aad2f0e384189872e794e35bbad29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:34:11.319893 21153 raft_consensus.cc:729] T e57c32124a154d2bbfccf97dcf62dcfa P fa9aad2f0e384189872e794e35bbad29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa9aad2f0e384189872e794e35bbad29, State: Initialized, Role: FOLLOWER
I20260926 16:34:11.320626 21153 consensus_queue.cc:260] T e57c32124a154d2bbfccf97dcf62dcfa P fa9aad2f0e384189872e794e35bbad29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ed5f76dd9e4a4593e96a1eaad01553" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 33857 } } peers { permanent_uuid: "ca053d21d9304c0aac25ce7dc6fdacf6" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45107 } } peers { permanent_uuid: "fa9aad2f0e384189872e794e35bbad29" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 34095 } }
I20260926 16:34:11.323374 21153 ts_tablet_manager.cc:1428] T e57c32124a154d2bbfccf97dcf62dcfa P fa9aad2f0e384189872e794e35bbad29: Time spent starting tablet: real 0.012s user 0.005s sys 0.007s
I20260926 16:34:11.324601 21155 raft_consensus.cc:348] T e57c32124a154d2bbfccf97dcf62dcfa P c6ed5f76dd9e4a4593e96a1eaad01553 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ed5f76dd9e4a4593e96a1eaad01553" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 33857 } } peers { permanent_uuid: "ca053d21d9304c0aac25ce7dc6fdacf6" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45107 } } peers { permanent_uuid: "fa9aad2f0e384189872e794e35bbad29" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 34095 } }
I20260926 16:34:11.325160 21155 raft_consensus.cc:374] T e57c32124a154d2bbfccf97dcf62dcfa P c6ed5f76dd9e4a4593e96a1eaad01553 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:34:11.325263 21155 raft_consensus.cc:729] T e57c32124a154d2bbfccf97dcf62dcfa P c6ed5f76dd9e4a4593e96a1eaad01553 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ed5f76dd9e4a4593e96a1eaad01553, State: Initialized, Role: FOLLOWER
I20260926 16:34:11.325960 21155 consensus_queue.cc:260] T e57c32124a154d2bbfccf97dcf62dcfa P c6ed5f76dd9e4a4593e96a1eaad01553 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ed5f76dd9e4a4593e96a1eaad01553" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 33857 } } peers { permanent_uuid: "ca053d21d9304c0aac25ce7dc6fdacf6" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45107 } } peers { permanent_uuid: "fa9aad2f0e384189872e794e35bbad29" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 34095 } }
I20260926 16:34:11.327215 21006 heartbeater.cc:503] Master 127.6.33.62:36283 was elected leader, sending a full tablet report...
I20260926 16:34:11.327943 21155 ts_tablet_manager.cc:1428] T e57c32124a154d2bbfccf97dcf62dcfa P c6ed5f76dd9e4a4593e96a1eaad01553: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260926 16:34:11.328275 21154 raft_consensus.cc:348] T e57c32124a154d2bbfccf97dcf62dcfa P ca053d21d9304c0aac25ce7dc6fdacf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ed5f76dd9e4a4593e96a1eaad01553" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 33857 } } peers { permanent_uuid: "ca053d21d9304c0aac25ce7dc6fdacf6" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45107 } } peers { permanent_uuid: "fa9aad2f0e384189872e794e35bbad29" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 34095 } }
I20260926 16:34:11.328802 21154 raft_consensus.cc:374] T e57c32124a154d2bbfccf97dcf62dcfa P ca053d21d9304c0aac25ce7dc6fdacf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:34:11.328951 21154 raft_consensus.cc:729] T e57c32124a154d2bbfccf97dcf62dcfa P ca053d21d9304c0aac25ce7dc6fdacf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca053d21d9304c0aac25ce7dc6fdacf6, State: Initialized, Role: FOLLOWER
I20260926 16:34:11.329588 21154 consensus_queue.cc:260] T e57c32124a154d2bbfccf97dcf62dcfa P ca053d21d9304c0aac25ce7dc6fdacf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ed5f76dd9e4a4593e96a1eaad01553" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 33857 } } peers { permanent_uuid: "ca053d21d9304c0aac25ce7dc6fdacf6" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45107 } } peers { permanent_uuid: "fa9aad2f0e384189872e794e35bbad29" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 34095 } }
I20260926 16:34:11.330770 21137 heartbeater.cc:503] Master 127.6.33.62:36283 was elected leader, sending a full tablet report...
I20260926 16:34:11.331373 21154 ts_tablet_manager.cc:1428] T e57c32124a154d2bbfccf97dcf62dcfa P ca053d21d9304c0aac25ce7dc6fdacf6: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20260926 16:34:11.346133 21161 raft_consensus.cc:482] T e57c32124a154d2bbfccf97dcf62dcfa P ca053d21d9304c0aac25ce7dc6fdacf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:34:11.346413 21161 raft_consensus.cc:504] T e57c32124a154d2bbfccf97dcf62dcfa P ca053d21d9304c0aac25ce7dc6fdacf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ed5f76dd9e4a4593e96a1eaad01553" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 33857 } } peers { permanent_uuid: "ca053d21d9304c0aac25ce7dc6fdacf6" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45107 } } peers { permanent_uuid: "fa9aad2f0e384189872e794e35bbad29" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 34095 } }
I20260926 16:34:11.348331 21161 leader_election.cc:288] T e57c32124a154d2bbfccf97dcf62dcfa P ca053d21d9304c0aac25ce7dc6fdacf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6ed5f76dd9e4a4593e96a1eaad01553 (127.6.33.2:33857), fa9aad2f0e384189872e794e35bbad29 (127.6.33.1:34095)
I20260926 16:34:11.354571 21139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:11.348493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.2:33857 (local address 127.6.33.3:48967)
0926 16:34:11.348780 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:11.348800 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 16:34:11.348937 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:34:11.349342 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:34:11.349347 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:34:11.349374 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:34:11.349587 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:11.349596 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:11.350605 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:11.350609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:11.352430 (+ 1821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:11.352445 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:11.353749 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:11.353755 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:11.353938 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:11.353962 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 16:34:11.354109 (+ 147us) client_negotiation.cc:240] Negotiation successful
0926 16:34:11.354251 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103}
I20260926 16:34:11.354571 21148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:11.348817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.3:48967 (local address 127.6.33.2:33857)
0926 16:34:11.349053 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:11.349061 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:11.349085 (+ 24us) server_negotiation.cc:407] Connection header received
0926 16:34:11.349151 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:11.349155 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:11.349212 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:11.349315 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:34:11.349737 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:11.350438 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:11.352646 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:11.353593 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:11.353739 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:11.353836 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:11.354134 (+ 298us) server_negotiation.cc:299] Negotiation successful
0926 16:34:11.354264 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 16:34:11.355410 21152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:11.348885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.3:56551 (local address 127.6.33.1:34095)
0926 16:34:11.349092 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:11.349100 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:11.349830 (+ 730us) server_negotiation.cc:407] Connection header received
0926 16:34:11.350074 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:11.350078 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:11.350147 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:11.350245 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:34:11.350804 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:11.351765 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:11.353706 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:11.354500 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:11.354658 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:11.354767 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:11.355037 (+ 270us) server_negotiation.cc:299] Negotiation successful
0926 16:34:11.355173 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 16:34:11.355490 21162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:11.348881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:34095 (local address 127.6.33.3:56551)
0926 16:34:11.349716 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:11.349730 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 16:34:11.349874 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:34:11.350273 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:34:11.350278 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:34:11.350300 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:34:11.350633 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:11.350643 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:11.351925 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:11.351930 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:11.353537 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:11.353550 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:11.354708 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:11.354714 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:11.354877 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:11.354900 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 16:34:11.355007 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 16:34:11.355134 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":716,"thread_start_us":474,"threads_started":1}
I20260926 16:34:11.355872 20961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57c32124a154d2bbfccf97dcf62dcfa" candidate_uuid: "ca053d21d9304c0aac25ce7dc6fdacf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ed5f76dd9e4a4593e96a1eaad01553" is_pre_election: true
I20260926 16:34:11.356491 20961 raft_consensus.cc:2454] T e57c32124a154d2bbfccf97dcf62dcfa P c6ed5f76dd9e4a4593e96a1eaad01553 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca053d21d9304c0aac25ce7dc6fdacf6 in term 0.
I20260926 16:34:11.356594 20829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57c32124a154d2bbfccf97dcf62dcfa" candidate_uuid: "ca053d21d9304c0aac25ce7dc6fdacf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa9aad2f0e384189872e794e35bbad29" is_pre_election: true
I20260926 16:34:11.357213 20829 raft_consensus.cc:2454] T e57c32124a154d2bbfccf97dcf62dcfa P fa9aad2f0e384189872e794e35bbad29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca053d21d9304c0aac25ce7dc6fdacf6 in term 0.
I20260926 16:34:11.357308 21025 leader_election.cc:302] T e57c32124a154d2bbfccf97dcf62dcfa P ca053d21d9304c0aac25ce7dc6fdacf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6ed5f76dd9e4a4593e96a1eaad01553, ca053d21d9304c0aac25ce7dc6fdacf6; no voters:
I20260926 16:34:11.357722 21161 raft_consensus.cc:2781] T e57c32124a154d2bbfccf97dcf62dcfa P ca053d21d9304c0aac25ce7dc6fdacf6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:34:11.357838 21161 raft_consensus.cc:482] T e57c32124a154d2bbfccf97dcf62dcfa P ca053d21d9304c0aac25ce7dc6fdacf6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:34:11.357924 21161 raft_consensus.cc:3037] T e57c32124a154d2bbfccf97dcf62dcfa P ca053d21d9304c0aac25ce7dc6fdacf6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:34:11.360093 21161 raft_consensus.cc:504] T e57c32124a154d2bbfccf97dcf62dcfa P ca053d21d9304c0aac25ce7dc6fdacf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ed5f76dd9e4a4593e96a1eaad01553" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 33857 } } peers { permanent_uuid: "ca053d21d9304c0aac25ce7dc6fdacf6" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45107 } } peers { permanent_uuid: "fa9aad2f0e384189872e794e35bbad29" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 34095 } }
I20260926 16:34:11.360721 21161 leader_election.cc:288] T e57c32124a154d2bbfccf97dcf62dcfa P ca053d21d9304c0aac25ce7dc6fdacf6 [CANDIDATE]: Term 1 election: Requested vote from peers c6ed5f76dd9e4a4593e96a1eaad01553 (127.6.33.2:33857), fa9aad2f0e384189872e794e35bbad29 (127.6.33.1:34095)
I20260926 16:34:11.361289 20829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57c32124a154d2bbfccf97dcf62dcfa" candidate_uuid: "ca053d21d9304c0aac25ce7dc6fdacf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa9aad2f0e384189872e794e35bbad29"
I20260926 16:34:11.361338 20961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57c32124a154d2bbfccf97dcf62dcfa" candidate_uuid: "ca053d21d9304c0aac25ce7dc6fdacf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ed5f76dd9e4a4593e96a1eaad01553"
I20260926 16:34:11.361522 20829 raft_consensus.cc:3037] T e57c32124a154d2bbfccf97dcf62dcfa P fa9aad2f0e384189872e794e35bbad29 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:34:11.361582 20961 raft_consensus.cc:3037] T e57c32124a154d2bbfccf97dcf62dcfa P c6ed5f76dd9e4a4593e96a1eaad01553 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:34:11.363852 20829 raft_consensus.cc:2454] T e57c32124a154d2bbfccf97dcf62dcfa P fa9aad2f0e384189872e794e35bbad29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca053d21d9304c0aac25ce7dc6fdacf6 in term 1.
I20260926 16:34:11.363922 20961 raft_consensus.cc:2454] T e57c32124a154d2bbfccf97dcf62dcfa P c6ed5f76dd9e4a4593e96a1eaad01553 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca053d21d9304c0aac25ce7dc6fdacf6 in term 1.
I20260926 16:34:11.364310 21028 leader_election.cc:302] T e57c32124a154d2bbfccf97dcf62dcfa P ca053d21d9304c0aac25ce7dc6fdacf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ca053d21d9304c0aac25ce7dc6fdacf6, fa9aad2f0e384189872e794e35bbad29; no voters:
I20260926 16:34:11.364668 21161 raft_consensus.cc:2781] T e57c32124a154d2bbfccf97dcf62dcfa P ca053d21d9304c0aac25ce7dc6fdacf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:34:11.365171 21161 raft_consensus.cc:686] T e57c32124a154d2bbfccf97dcf62dcfa P ca053d21d9304c0aac25ce7dc6fdacf6 [term 1 LEADER]: Becoming Leader. State: Replica: ca053d21d9304c0aac25ce7dc6fdacf6, State: Running, Role: LEADER
I20260926 16:34:11.365662 21161 consensus_queue.cc:237] T e57c32124a154d2bbfccf97dcf62dcfa P ca053d21d9304c0aac25ce7dc6fdacf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ed5f76dd9e4a4593e96a1eaad01553" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 33857 } } peers { permanent_uuid: "ca053d21d9304c0aac25ce7dc6fdacf6" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45107 } } peers { permanent_uuid: "fa9aad2f0e384189872e794e35bbad29" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 34095 } }
I20260926 16:34:11.370893 20688 catalog_manager.cc:5495] T e57c32124a154d2bbfccf97dcf62dcfa P ca053d21d9304c0aac25ce7dc6fdacf6 reported cstate change: term changed from 0 to 1, leader changed from <none> to ca053d21d9304c0aac25ce7dc6fdacf6 (127.6.33.3). New cstate: current_term: 1 leader_uuid: "ca053d21d9304c0aac25ce7dc6fdacf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ed5f76dd9e4a4593e96a1eaad01553" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 33857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca053d21d9304c0aac25ce7dc6fdacf6" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa9aad2f0e384189872e794e35bbad29" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 34095 } health_report { overall_health: UNKNOWN } } }
I20260926 16:34:11.409790 21150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:11.404284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44704 (local address 127.6.33.3:45107)
0926 16:34:11.404522 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:11.404549 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:11.404569 (+ 20us) server_negotiation.cc:407] Connection header received
0926 16:34:11.404662 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:11.404666 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:11.404784 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:11.404923 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:34:11.405274 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:11.406016 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:11.407536 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:11.407945 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:11.408055 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:11.408605 (+ 550us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:34:11.409224 (+ 619us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:34:11.409337 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:11.409488 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 16:34:11.409584 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260926 16:34:11.418674 20829 raft_consensus.cc:1261] T e57c32124a154d2bbfccf97dcf62dcfa P fa9aad2f0e384189872e794e35bbad29 [term 1 FOLLOWER]: Refusing update from remote peer ca053d21d9304c0aac25ce7dc6fdacf6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:34:11.418668 20961 raft_consensus.cc:1261] T e57c32124a154d2bbfccf97dcf62dcfa P c6ed5f76dd9e4a4593e96a1eaad01553 [term 1 FOLLOWER]: Refusing update from remote peer ca053d21d9304c0aac25ce7dc6fdacf6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:34:11.419756 21161 consensus_queue.cc:1035] T e57c32124a154d2bbfccf97dcf62dcfa P ca053d21d9304c0aac25ce7dc6fdacf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa9aad2f0e384189872e794e35bbad29" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:34:11.420063 21163 consensus_queue.cc:1035] T e57c32124a154d2bbfccf97dcf62dcfa P ca053d21d9304c0aac25ce7dc6fdacf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ed5f76dd9e4a4593e96a1eaad01553" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 33857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:34:11.433018 21167 mvcc.cc:205] Tried to move back new op lower bound from 7333644088999698432 to 7333644088797642752. Current Snapshot: MvccSnapshot[applied={T|T < 7333644088999698432}]
I20260926 16:34:11.437173 21166 mvcc.cc:205] Tried to move back new op lower bound from 7333644088999698432 to 7333644088797642752. Current Snapshot: MvccSnapshot[applied={T|T < 7333644088999698432}]
I20260926 16:34:11.446877 20689 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:57862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:34:11.447213 20689 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:57862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:34:11.451107 20689 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1681ae7327bb4f84a8620eb060da18ce: Sending DeleteTablet for 3 replicas of tablet e57c32124a154d2bbfccf97dcf62dcfa
I20260926 16:34:11.452345 20941 tablet_service.cc:1514] Processing DeleteTablet for tablet e57c32124a154d2bbfccf97dcf62dcfa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:34:11 UTC) from {username='kudu', principal='kudu/127.6.33.62@KRBTEST.COM'} at 127.0.0.1:60926
I20260926 16:34:11.452409 21072 tablet_service.cc:1514] Processing DeleteTablet for tablet e57c32124a154d2bbfccf97dcf62dcfa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:34:11 UTC) from {username='kudu', principal='kudu/127.6.33.62@KRBTEST.COM'} at 127.0.0.1:44702
I20260926 16:34:11.452409 20809 tablet_service.cc:1514] Processing DeleteTablet for tablet e57c32124a154d2bbfccf97dcf62dcfa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:34:11 UTC) from {username='kudu', principal='kudu/127.6.33.62@KRBTEST.COM'} at 127.0.0.1:53564
I20260926 16:34:11.455219 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 20726
I20260926 16:34:11.467850 21177 tablet_replica.cc:330] T e57c32124a154d2bbfccf97dcf62dcfa P c6ed5f76dd9e4a4593e96a1eaad01553: stopping tablet replica
I20260926 16:34:11.468408 21177 raft_consensus.cc:2229] T e57c32124a154d2bbfccf97dcf62dcfa P c6ed5f76dd9e4a4593e96a1eaad01553 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:34:11.469146 21177 raft_consensus.cc:2258] T e57c32124a154d2bbfccf97dcf62dcfa P c6ed5f76dd9e4a4593e96a1eaad01553 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:34:11.471453 21177 ts_tablet_manager.cc:1905] T e57c32124a154d2bbfccf97dcf62dcfa P c6ed5f76dd9e4a4593e96a1eaad01553: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:34:11.479211 21177 ts_tablet_manager.cc:1918] T e57c32124a154d2bbfccf97dcf62dcfa P c6ed5f76dd9e4a4593e96a1eaad01553: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:34:11.479425 21177 log.cc:1196] T e57c32124a154d2bbfccf97dcf62dcfa P c6ed5f76dd9e4a4593e96a1eaad01553: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-1/wal/wals/e57c32124a154d2bbfccf97dcf62dcfa
I20260926 16:34:11.480044 21177 ts_tablet_manager.cc:1939] T e57c32124a154d2bbfccf97dcf62dcfa P c6ed5f76dd9e4a4593e96a1eaad01553: Deleting consensus metadata
I20260926 16:34:11.482255 20673 catalog_manager.cc:4841] TS c6ed5f76dd9e4a4593e96a1eaad01553 (127.6.33.2:33857): tablet e57c32124a154d2bbfccf97dcf62dcfa (table test-table [id=dfc45ca7622847be80f35776a8066351]) successfully deleted
I20260926 16:34:11.485425 21176 tablet_replica.cc:330] T e57c32124a154d2bbfccf97dcf62dcfa P ca053d21d9304c0aac25ce7dc6fdacf6: stopping tablet replica
I20260926 16:34:11.486167 21176 raft_consensus.cc:2229] T e57c32124a154d2bbfccf97dcf62dcfa P ca053d21d9304c0aac25ce7dc6fdacf6 [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:34:11.487587 21175 tablet_replica.cc:330] T e57c32124a154d2bbfccf97dcf62dcfa P fa9aad2f0e384189872e794e35bbad29: stopping tablet replica
I20260926 16:34:11.488148 21175 raft_consensus.cc:2229] T e57c32124a154d2bbfccf97dcf62dcfa P fa9aad2f0e384189872e794e35bbad29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:34:11.488476 21152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:11.456948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53572 (local address 127.6.33.1:34095)
0926 16:34:11.457184 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:11.457193 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:11.457218 (+ 25us) server_negotiation.cc:407] Connection header received
0926 16:34:11.457283 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:11.457289 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:11.457416 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:11.457538 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:34:11.460066 (+ 2528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:11.461723 (+ 1657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:11.464597 (+ 2874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:11.465589 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:11.465753 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:11.474516 (+ 8763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:34:11.474546 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:34:11.474574 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:34:11.474651 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:34:11.481098 (+ 6447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:11.483879 (+ 2781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:11.483896 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:11.483903 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:11.484188 (+ 285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:11.485022 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:11.485031 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:11.485038 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:11.486007 (+ 969us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:34:11.487251 (+ 1244us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:11.487559 (+ 308us) server_negotiation.cc:299] Negotiation successful
0926 16:34:11.487956 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 16:34:11.488737 21175 raft_consensus.cc:2258] T e57c32124a154d2bbfccf97dcf62dcfa P fa9aad2f0e384189872e794e35bbad29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:34:11.488481 21176 raft_consensus.cc:2258] T e57c32124a154d2bbfccf97dcf62dcfa P ca053d21d9304c0aac25ce7dc6fdacf6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:34:11.489514 20869 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:34:11.491927 21176 ts_tablet_manager.cc:1905] T e57c32124a154d2bbfccf97dcf62dcfa P ca053d21d9304c0aac25ce7dc6fdacf6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:34:11.499881 21176 ts_tablet_manager.cc:1918] T e57c32124a154d2bbfccf97dcf62dcfa P ca053d21d9304c0aac25ce7dc6fdacf6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:34:11.500097 21176 log.cc:1196] T e57c32124a154d2bbfccf97dcf62dcfa P ca053d21d9304c0aac25ce7dc6fdacf6: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-2/wal/wals/e57c32124a154d2bbfccf97dcf62dcfa
I20260926 16:34:11.500587 21176 ts_tablet_manager.cc:1939] T e57c32124a154d2bbfccf97dcf62dcfa P ca053d21d9304c0aac25ce7dc6fdacf6: Deleting consensus metadata
I20260926 16:34:11.502282 20675 catalog_manager.cc:4841] TS ca053d21d9304c0aac25ce7dc6fdacf6 (127.6.33.3:45107): tablet e57c32124a154d2bbfccf97dcf62dcfa (table test-table [id=dfc45ca7622847be80f35776a8066351]) successfully deleted
I20260926 16:34:11.604808 21175 ts_tablet_manager.cc:1905] T e57c32124a154d2bbfccf97dcf62dcfa P fa9aad2f0e384189872e794e35bbad29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:34:11.606178 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 20726
I20260926 16:34:11.630757 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 20878
W20260926 16:34:11.631351 20676 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.33.1:34095 (error 108)
I20260926 16:34:11.634198 21149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:11.633642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:34095 (local address 127.0.0.1:53576)
0926 16:34:11.633911 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:11.634047 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.1:34095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
W20260926 16:34:11.634414 20676 catalog_manager.cc:4584] TS fa9aad2f0e384189872e794e35bbad29 (127.6.33.1:34095): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e57c32124a154d2bbfccf97dcf62dcfa: Network error: Client connection negotiation failed: client connection to 127.6.33.1:34095: connect: Connection refused (error 111)
I20260926 16:34:11.650345 21148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:11.633038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60940 (local address 127.6.33.2:33857)
0926 16:34:11.633281 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:11.633287 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:11.634626 (+ 1339us) server_negotiation.cc:407] Connection header received
0926 16:34:11.634918 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:11.634926 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:11.635031 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:11.635153 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:34:11.636521 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:11.637795 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:11.639207 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:11.639843 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:11.639974 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:11.642633 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:34:11.642652 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:34:11.642674 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:34:11.642715 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:34:11.647051 (+ 4336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:11.648094 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:11.648102 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:11.648104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:11.648299 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:11.648732 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:11.648735 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:11.648737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:11.649281 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:34:11.649495 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:11.649896 (+ 401us) server_negotiation.cc:299] Negotiation successful
0926 16:34:11.650057 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 16:34:11.651042 21001 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:34:11.688417 21149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:11.687824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:34095 (local address 127.0.0.1:53588)
0926 16:34:11.688056 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:11.688220 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.1:34095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260926 16:34:11.689675 21149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:11.689249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:34095 (local address 127.0.0.1:53594)
0926 16:34:11.689419 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:11.689532 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.1:34095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 16:34:11.750566 21149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:11.749992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:34095 (local address 127.0.0.1:53610)
0926 16:34:11.750242 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:11.750353 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.1:34095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260926 16:34:11.751791 21149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:11.751363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:34095 (local address 127.0.0.1:53618)
0926 16:34:11.751544 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:11.751641 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.1:34095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 16:34:11.757804 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 20878
I20260926 16:34:11.775696 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 21009
I20260926 16:34:11.791771 21150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:11.776728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44718 (local address 127.6.33.3:45107)
0926 16:34:11.776972 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:11.776980 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:11.777002 (+ 22us) server_negotiation.cc:407] Connection header received
0926 16:34:11.777085 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:11.777089 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:11.777148 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:11.777265 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:34:11.778686 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:11.779537 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:11.781142 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:11.781636 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:11.781785 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:11.784616 (+ 2831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:34:11.784632 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:34:11.784648 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:34:11.784694 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:34:11.788310 (+ 3616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:11.789316 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:11.789324 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:11.789326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:11.789500 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:11.789926 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:11.789936 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:11.789938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:11.790323 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:34:11.790684 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:11.791013 (+ 329us) server_negotiation.cc:299] Negotiation successful
0926 16:34:11.791264 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260926 16:34:11.792900 21132 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:34:11.832435 21149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:11.831933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:34095 (local address 127.0.0.1:53622)
0926 16:34:11.832182 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:11.832273 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.1:34095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260926 16:34:11.833729 21149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:11.833246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:34095 (local address 127.0.0.1:53624)
0926 16:34:11.833510 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:11.833598 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.1:34095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
I20260926 16:34:11.919179 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 21009
I20260926 16:34:11.939405 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 20656
I20260926 16:34:11.940593 20719 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:34:12.058193 21149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:12.057425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:34095 (local address 127.0.0.1:53632)
0926 16:34:12.057786 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:12.057937 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.1:34095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204}
I20260926 16:34:12.058187 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 20656
I20260926 16:34:12.081166 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:36283
--webserver_interface=127.6.33.62
--webserver_port=46143
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:44077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.33.62:36283
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:34:12.417037 21183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:34:12.417346 21183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:34:12.417528 21183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:34:12.425190 21183 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:34:12.425328 21183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:34:12.425375 21183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:34:12.425508 21183 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:34:12.425572 21183 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:34:12.435928 21183 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:44077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.33.62:36283
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:36283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.33.62
--webserver_port=46143
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:34:12.437490 21183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:34:12.438953 21183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:34:12.451787 21189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:34:12.451887 21188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:34:12.451887 21191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:34:12.453270 21183 server_base.cc:1034] running on GCE node
I20260926 16:34:12.454188 21183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:34:12.455945 21183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:34:12.457180 21183 hybrid_clock.cc:648] HybridClock initialized: now 1790440452457131 us; error 43 us; skew 500 ppm
I20260926 16:34:12.457549 21183 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:34:12 dist-test-slave-khd3 krb5kdc[20106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440452, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:34:12.461043 21183 init.cc:377] Logged in from keytab as kudu/127.6.33.62@KRBTEST.COM (short username kudu)
I20260926 16:34:12.462615 21183 webserver.cc:460] Webserver started at http://127.6.33.62:46143/ using document root <none> and password file <none>
I20260926 16:34:12.463366 21183 fs_manager.cc:359] Metadata directory not provided
I20260926 16:34:12.463553 21183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:34:12.469411 21183 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 16:34:12.472457 21198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:34:12.473953 21183 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:34:12.474201 21183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/master-0/wal
uuid: "1681ae7327bb4f84a8620eb060da18ce"
format_stamp: "Formatted at 2026-09-26 16:34:06 on dist-test-slave-khd3"
server_key: "31d371d57d2a98fc3e204258b17c14d6"
server_key_iv: "b7c4e0b0d51311bcbb459f9afb5d10a9"
server_key_version: "encryptionkey@0"
I20260926 16:34:12.475237 21183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:34:12.498116 21183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:34:12.502342 21183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:34:12.502643 21183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:34:12.522637 21183 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.62:36283
I20260926 16:34:12.522677 21249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.62:36283 every 8 connection(s)
I20260926 16:34:12.524245 21183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
I20260926 16:34:12.524919 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 21183
I20260926 16:34:12.525870 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.1:34095
--local_ip_for_outbound_sockets=127.6.33.1
--tserver_master_addrs=127.6.33.62:36283
--webserver_port=41611
--webserver_interface=127.6.33.1
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab
--principal=kudu/127.6.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:44077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:34:12.531306 21250 sys_catalog.cc:263] Verifying existing consensus state
I20260926 16:34:12.534592 21250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1681ae7327bb4f84a8620eb060da18ce: Bootstrap starting.
I20260926 16:34:12.555075 21250 log.cc:824] T 00000000000000000000000000000000 P 1681ae7327bb4f84a8620eb060da18ce: Log is configured to *not* fsync() on all Append() calls
I20260926 16:34:12.571229 21250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1681ae7327bb4f84a8620eb060da18ce: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 16:34:12.571817 21250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1681ae7327bb4f84a8620eb060da18ce: Bootstrap complete.
I20260926 16:34:12.581763 21250 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1681ae7327bb4f84a8620eb060da18ce [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1681ae7327bb4f84a8620eb060da18ce" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 36283 } }
I20260926 16:34:12.582695 21250 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1681ae7327bb4f84a8620eb060da18ce [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1681ae7327bb4f84a8620eb060da18ce, State: Initialized, Role: FOLLOWER
I20260926 16:34:12.583290 21250 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1681ae7327bb4f84a8620eb060da18ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1681ae7327bb4f84a8620eb060da18ce" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 36283 } }
I20260926 16:34:12.583531 21250 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1681ae7327bb4f84a8620eb060da18ce [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:34:12.583683 21250 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1681ae7327bb4f84a8620eb060da18ce [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:34:12.583873 21250 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1681ae7327bb4f84a8620eb060da18ce [term 2 FOLLOWER]: Advancing to term 3
I20260926 16:34:12.588037 21250 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1681ae7327bb4f84a8620eb060da18ce [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1681ae7327bb4f84a8620eb060da18ce" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 36283 } }
I20260926 16:34:12.588532 21250 leader_election.cc:302] T 00000000000000000000000000000000 P 1681ae7327bb4f84a8620eb060da18ce [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1681ae7327bb4f84a8620eb060da18ce; no voters:
I20260926 16:34:12.589294 21250 leader_election.cc:288] T 00000000000000000000000000000000 P 1681ae7327bb4f84a8620eb060da18ce [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 16:34:12.589434 21254 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1681ae7327bb4f84a8620eb060da18ce [term 3 FOLLOWER]: Leader election won for term 3
I20260926 16:34:12.590632 21254 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1681ae7327bb4f84a8620eb060da18ce [term 3 LEADER]: Becoming Leader. State: Replica: 1681ae7327bb4f84a8620eb060da18ce, State: Running, Role: LEADER
I20260926 16:34:12.591226 21254 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1681ae7327bb4f84a8620eb060da18ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1681ae7327bb4f84a8620eb060da18ce" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 36283 } }
I20260926 16:34:12.591518 21250 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1681ae7327bb4f84a8620eb060da18ce [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:34:12.594349 21255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1681ae7327bb4f84a8620eb060da18ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "1681ae7327bb4f84a8620eb060da18ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1681ae7327bb4f84a8620eb060da18ce" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 36283 } } }
I20260926 16:34:12.594787 21255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1681ae7327bb4f84a8620eb060da18ce [sys.catalog]: This master's current role is: LEADER
I20260926 16:34:12.597368 21256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1681ae7327bb4f84a8620eb060da18ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1681ae7327bb4f84a8620eb060da18ce. Latest consensus state: current_term: 3 leader_uuid: "1681ae7327bb4f84a8620eb060da18ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1681ae7327bb4f84a8620eb060da18ce" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 36283 } } }
I20260926 16:34:12.597743 21256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1681ae7327bb4f84a8620eb060da18ce [sys.catalog]: This master's current role is: LEADER
I20260926 16:34:12.599365 21260 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:34:12.608451 21260 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:34:12.610471 21260 catalog_manager.cc:1259] Loaded cluster ID: 4a47e9ca2a544b0aa9e2c015c3a18ef4
I20260926 16:34:12.610656 21260 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260926 16:34:12.612946 21260 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f43533569ad google::LogMessage::Fail() at ??:0
@ 0x7f435335988a google::LogMessage::SendToLog() at ??:0
@ 0x7f43533564de google::LogMessage::Flush() at ??:0
@ 0x7f4353357a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f4368d89512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f4368d86976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f4368e0e610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f4368e0e439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f4355fa2345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f4355fa591c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f4355fa5799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f4355f80eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f435925a6db start_thread at ??:0
@ 0x7f43521f061f clone at ??:0
W20260926 16:34:12.895717 21252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:34:12.896097 21252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:34:12.896267 21252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:34:12.903877 21252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:34:12.904067 21252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:34:12.904245 21252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.1
I20260926 16:34:12.914427 21252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:44077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab
--principal=kudu/127.6.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.1:34095
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.33.1
--webserver_port=41611
--tserver_master_addrs=127.6.33.62:36283
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.1
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:34:12.915869 21252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:34:12.917290 21252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:34:12.930436 21292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:34:12.930531 21294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:34:12.930436 21291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:34:12.931267 21252 server_base.cc:1034] running on GCE node
I20260926 16:34:12.932013 21252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:34:12.933106 21252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:34:12.934352 21252 hybrid_clock.cc:648] HybridClock initialized: now 1790440452934301 us; error 46 us; skew 500 ppm
I20260926 16:34:12.934692 21252 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:34:12 dist-test-slave-khd3 krb5kdc[20106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440452, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:34:12.937942 21252 init.cc:377] Logged in from keytab as kudu/127.6.33.1@KRBTEST.COM (short username kudu)
I20260926 16:34:12.939505 21252 webserver.cc:460] Webserver started at http://127.6.33.1:41611/ using document root <none> and password file <none>
I20260926 16:34:12.940263 21252 fs_manager.cc:359] Metadata directory not provided
I20260926 16:34:12.940405 21252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:34:12.945879 21252 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 16:34:12.950883 21301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:34:12.952323 21252 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 16:34:12.952512 21252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-0/wal
uuid: "fa9aad2f0e384189872e794e35bbad29"
format_stamp: "Formatted at 2026-09-26 16:34:07 on dist-test-slave-khd3"
server_key: "0a324ffecaed301b7df470ea749a8246"
server_key_iv: "9e692dee5188eb253425075ecc5d2f6c"
server_key_version: "encryptionkey@0"
I20260926 16:34:12.953925 21252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:34:12.978520 21252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:34:12.983043 21252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:34:12.983336 21252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:34:12.984380 21252 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:34:12.986438 21308 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260926 16:34:12.992796 21252 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 16:34:12.992956 21252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.000s sys 0.001s
I20260926 16:34:12.993141 21252 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 16:34:12.996296 21252 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 16:34:12.996409 21252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20260926 16:34:12.996812 21308 tablet_bootstrap.cc:492] T e57c32124a154d2bbfccf97dcf62dcfa P fa9aad2f0e384189872e794e35bbad29: Bootstrap starting.
I20260926 16:34:13.005621 21308 log.cc:824] T e57c32124a154d2bbfccf97dcf62dcfa P fa9aad2f0e384189872e794e35bbad29: Log is configured to *not* fsync() on all Append() calls
I20260926 16:34:13.023329 21308 tablet_bootstrap.cc:492] T e57c32124a154d2bbfccf97dcf62dcfa P fa9aad2f0e384189872e794e35bbad29: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 16:34:13.024209 21308 tablet_bootstrap.cc:492] T e57c32124a154d2bbfccf97dcf62dcfa P fa9aad2f0e384189872e794e35bbad29: Bootstrap complete.
I20260926 16:34:13.024729 21308 ts_tablet_manager.cc:1397] T e57c32124a154d2bbfccf97dcf62dcfa P fa9aad2f0e384189872e794e35bbad29: Time spent bootstrapping tablet: real 0.028s user 0.021s sys 0.006s
I20260926 16:34:13.034063 21308 raft_consensus.cc:348] T e57c32124a154d2bbfccf97dcf62dcfa P fa9aad2f0e384189872e794e35bbad29 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ed5f76dd9e4a4593e96a1eaad01553" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 33857 } } peers { permanent_uuid: "ca053d21d9304c0aac25ce7dc6fdacf6" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45107 } } peers { permanent_uuid: "fa9aad2f0e384189872e794e35bbad29" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 34095 } }
I20260926 16:34:13.035383 21308 raft_consensus.cc:729] T e57c32124a154d2bbfccf97dcf62dcfa P fa9aad2f0e384189872e794e35bbad29 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa9aad2f0e384189872e794e35bbad29, State: Initialized, Role: FOLLOWER
I20260926 16:34:13.036137 21308 consensus_queue.cc:260] T e57c32124a154d2bbfccf97dcf62dcfa P fa9aad2f0e384189872e794e35bbad29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ed5f76dd9e4a4593e96a1eaad01553" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 33857 } } peers { permanent_uuid: "ca053d21d9304c0aac25ce7dc6fdacf6" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45107 } } peers { permanent_uuid: "fa9aad2f0e384189872e794e35bbad29" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 34095 } }
I20260926 16:34:13.038282 21308 ts_tablet_manager.cc:1428] T e57c32124a154d2bbfccf97dcf62dcfa P fa9aad2f0e384189872e794e35bbad29: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260926 16:34:13.063354 21252 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.1:34095
I20260926 16:34:13.063373 21416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.1:34095 every 8 connection(s)
I20260926 16:34:13.064658 21252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
I20260926 16:34:13.069857 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 21252
I20260926 16:34:13.069885 21419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:13.067797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36283 (local address 127.6.33.1:41465)
0926 16:34:13.068661 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:13.068935 (+ 274us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":378,"threads_started":1}
I20260926 16:34:13.070526 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.2:33857
--local_ip_for_outbound_sockets=127.6.33.2
--tserver_master_addrs=127.6.33.62:36283
--webserver_port=38415
--webserver_interface=127.6.33.2
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab
--principal=kudu/127.6.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:44077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:34:13.070690 21417 heartbeater.cc:644] Failed to heartbeat to 127.6.33.62:36283 (0 consecutive failures): Network error: Failed to ping master at 127.6.33.62:36283: Client connection negotiation failed: client connection to 127.6.33.62:36283: connect: Connection refused (error 111)
I20260926 16:34:13.071532 21419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:13.071202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36283 (local address 127.6.33.1:54277)
0926 16:34:13.071354 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:13.071433 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 16:34:13.072347 21419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:13.072015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36283 (local address 127.6.33.1:33807)
0926 16:34:13.072158 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:13.072237 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 16:34:13.072549 21417 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 16:34:13.402863 21420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:34:13.403209 21420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:34:13.403385 21420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:34:13.410676 21420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:34:13.410830 21420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:34:13.411016 21420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.2
I20260926 16:34:13.420799 21420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:44077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab
--principal=kudu/127.6.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.2:33857
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.33.2
--webserver_port=38415
--tserver_master_addrs=127.6.33.62:36283
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.2
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:34:13.422263 21420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:34:13.423606 21420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:34:13.437204 21425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:34:13.437148 21426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:34:13.437201 21428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:34:13.437870 21420 server_base.cc:1034] running on GCE node
I20260926 16:34:13.438563 21420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:34:13.439671 21420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:34:13.440871 21420 hybrid_clock.cc:648] HybridClock initialized: now 1790440453440819 us; error 45 us; skew 500 ppm
I20260926 16:34:13.441207 21420 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:34:13 dist-test-slave-khd3 krb5kdc[20106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440453, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:34:13.444593 21420 init.cc:377] Logged in from keytab as kudu/127.6.33.2@KRBTEST.COM (short username kudu)
I20260926 16:34:13.446391 21420 webserver.cc:460] Webserver started at http://127.6.33.2:38415/ using document root <none> and password file <none>
I20260926 16:34:13.447121 21420 fs_manager.cc:359] Metadata directory not provided
I20260926 16:34:13.447269 21420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:34:13.453019 21420 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 16:34:13.456097 21435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:34:13.457760 21420 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:34:13.457958 21420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-1/wal
uuid: "c6ed5f76dd9e4a4593e96a1eaad01553"
format_stamp: "Formatted at 2026-09-26 16:34:07 on dist-test-slave-khd3"
server_key: "cd5beecea99a70c7b4a489beb613869e"
server_key_iv: "5a67ea834ccffb23601ab2c028fdf783"
server_key_version: "encryptionkey@0"
I20260926 16:34:13.459088 21420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:34:13.475579 21420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:34:13.479732 21420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:34:13.480023 21420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:34:13.481072 21420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:34:13.482438 21420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:34:13.482559 21420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:34:13.482719 21420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:34:13.482817 21420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:34:13.517537 21420 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.2:33857
I20260926 16:34:13.517673 21547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.2:33857 every 8 connection(s)
I20260926 16:34:13.519122 21420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
I20260926 16:34:13.523861 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 21420
I20260926 16:34:13.524444 21550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:13.522369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36283 (local address 127.6.33.2:47387)
0926 16:34:13.523122 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:13.523349 (+ 227us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":296,"threads_started":1}
I20260926 16:34:13.524490 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.3:45107
--local_ip_for_outbound_sockets=127.6.33.3
--tserver_master_addrs=127.6.33.62:36283
--webserver_port=36299
--webserver_interface=127.6.33.3
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab
--principal=kudu/127.6.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:44077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:34:13.525296 21548 heartbeater.cc:644] Failed to heartbeat to 127.6.33.62:36283 (0 consecutive failures): Network error: Failed to ping master at 127.6.33.62:36283: Client connection negotiation failed: client connection to 127.6.33.62:36283: connect: Connection refused (error 111)
I20260926 16:34:13.526315 21550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:13.525970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36283 (local address 127.6.33.2:40803)
0926 16:34:13.526135 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:13.526206 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 16:34:13.527204 21550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:13.526862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36283 (local address 127.6.33.2:38749)
0926 16:34:13.527007 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:13.527096 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 16:34:13.527417 21548 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 16:34:13.865474 21551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:34:13.865756 21551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:34:13.865912 21551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:34:13.873499 21551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:34:13.873611 21551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:34:13.873814 21551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.3
I20260926 16:34:13.883677 21551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:44077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab
--principal=kudu/127.6.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.3:45107
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.33.3
--webserver_port=36299
--tserver_master_addrs=127.6.33.62:36283
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.3
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:34:13.885076 21551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:34:13.886395 21551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:34:13.899762 21556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:34:13.899842 21559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:34:13.899986 21557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:34:13.901062 21551 server_base.cc:1034] running on GCE node
I20260926 16:34:13.901690 21551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:34:13.902738 21551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:34:13.903964 21551 hybrid_clock.cc:648] HybridClock initialized: now 1790440453903919 us; error 68 us; skew 500 ppm
I20260926 16:34:13.904299 21551 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:34:13 dist-test-slave-khd3 krb5kdc[20106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440453, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:34:13.907512 21551 init.cc:377] Logged in from keytab as kudu/127.6.33.3@KRBTEST.COM (short username kudu)
I20260926 16:34:13.909034 21551 webserver.cc:460] Webserver started at http://127.6.33.3:36299/ using document root <none> and password file <none>
I20260926 16:34:13.909683 21551 fs_manager.cc:359] Metadata directory not provided
I20260926 16:34:13.909781 21551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:34:13.915109 21551 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260926 16:34:13.918000 21566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:34:13.919373 21551 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 16:34:13.919534 21551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-2/wal
uuid: "ca053d21d9304c0aac25ce7dc6fdacf6"
format_stamp: "Formatted at 2026-09-26 16:34:08 on dist-test-slave-khd3"
server_key: "95608f2856ca05c3a623b28ff261dea6"
server_key_iv: "3cebd328b8ea9a153efe2e27eccbd748"
server_key_version: "encryptionkey@0"
I20260926 16:34:13.920540 21551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:34:13.933939 21551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:34:13.938246 21551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:34:13.938516 21551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:34:13.939445 21551 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:34:13.940747 21551 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:34:13.940891 21551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:34:13.941045 21551 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:34:13.941125 21551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:34:13.975765 21551 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.3:45107
I20260926 16:34:13.975875 21678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.3:45107 every 8 connection(s)
I20260926 16:34:13.977381 21551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
I20260926 16:34:13.978076 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 21551
I20260926 16:34:13.983191 21682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:13.981189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36283 (local address 127.6.33.3:36771)
0926 16:34:13.982017 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:13.982226 (+ 209us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":302,"threads_started":1}
W20260926 16:34:13.983996 21679 heartbeater.cc:644] Failed to heartbeat to 127.6.33.62:36283 (0 consecutive failures): Network error: Failed to ping master at 127.6.33.62:36283: Client connection negotiation failed: client connection to 127.6.33.62:36283: connect: Connection refused (error 111)
I20260926 16:34:13.984966 21682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:13.984579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36283 (local address 127.6.33.3:40105)
0926 16:34:13.984732 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:13.984860 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 16:34:13.985965 21682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:13.985657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36283 (local address 127.6.33.3:38367)
0926 16:34:13.985782 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:13.985856 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 16:34:13.986330 21679 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 16:34:14.073773 21683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:14.073085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36283 (local address 127.6.33.1:46219)
0926 16:34:14.073505 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:14.073592 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":202,"threads_started":1}
I20260926 16:34:14.528972 21684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:14.528035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36283 (local address 127.6.33.2:51839)
0926 16:34:14.528508 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:14.528623 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":244,"threads_started":1}
I20260926 16:34:14.552901 21685 raft_consensus.cc:482] T e57c32124a154d2bbfccf97dcf62dcfa P fa9aad2f0e384189872e794e35bbad29 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:34:14.553232 21685 raft_consensus.cc:504] T e57c32124a154d2bbfccf97dcf62dcfa P fa9aad2f0e384189872e794e35bbad29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ed5f76dd9e4a4593e96a1eaad01553" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 33857 } } peers { permanent_uuid: "ca053d21d9304c0aac25ce7dc6fdacf6" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45107 } } peers { permanent_uuid: "fa9aad2f0e384189872e794e35bbad29" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 34095 } }
I20260926 16:34:14.554993 21685 leader_election.cc:288] T e57c32124a154d2bbfccf97dcf62dcfa P fa9aad2f0e384189872e794e35bbad29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c6ed5f76dd9e4a4593e96a1eaad01553 (127.6.33.2:33857), ca053d21d9304c0aac25ce7dc6fdacf6 (127.6.33.3:45107)
Sep 26 16:34:14 dist-test-slave-khd3 krb5kdc[20106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440452, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.1@KRBTEST.COM for kudu/127.6.33.2@KRBTEST.COM
Sep 26 16:34:14 dist-test-slave-khd3 krb5kdc[20106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440452, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.1@KRBTEST.COM for kudu/127.6.33.3@KRBTEST.COM
I20260926 16:34:14.579988 21686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:14.555332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.3:45107 (local address 127.6.33.1:42241)
0926 16:34:14.556088 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:14.556113 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 16:34:14.556669 (+ 556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:34:14.560535 (+ 3866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:34:14.560540 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:34:14.561123 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:34:14.561472 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:14.561493 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:14.563943 (+ 2450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:14.563948 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:14.565463 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:14.565476 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:14.566551 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:14.566558 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:14.566825 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:14.567676 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:34:14.567693 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:34:14.570620 (+ 2927us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:34:14.575737 (+ 5117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:34:14.575746 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:34:14.575751 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:34:14.576323 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:34:14.576883 (+ 560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:34:14.576888 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:34:14.576891 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:34:14.577071 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:34:14.577970 (+ 899us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:34:14.577979 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:34:14.578538 (+ 559us) client_negotiation.cc:769] Sending connection context
0926 16:34:14.578898 (+ 360us) client_negotiation.cc:240] Negotiation successful
0926 16:34:14.579527 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"mutex_wait_us":208,"spinlock_wait_cycles":2176,"thread_start_us":407,"threads_started":1}
I20260926 16:34:14.580029 21688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:14.556505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:42241 (local address 127.6.33.3:45107)
0926 16:34:14.557091 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:14.557108 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:14.557148 (+ 40us) server_negotiation.cc:407] Connection header received
0926 16:34:14.559323 (+ 2175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:14.559335 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:14.559790 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:14.560136 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:34:14.561684 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:14.563706 (+ 2022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:14.565615 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:14.566282 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:14.566521 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:14.570909 (+ 4388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:34:14.570921 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:34:14.570942 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:34:14.571000 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:34:14.575583 (+ 4583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:14.576469 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:14.576477 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:14.576485 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:14.576712 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:14.577185 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:14.577189 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:14.577192 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:14.577801 (+ 609us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:34:14.578004 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:14.579164 (+ 1160us) server_negotiation.cc:299] Negotiation successful
0926 16:34:14.579606 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":276,"threads_started":1}
I20260926 16:34:14.580029 21687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:14.555623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:44743 (local address 127.6.33.2:33857)
0926 16:34:14.556440 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:14.556458 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:14.556498 (+ 40us) server_negotiation.cc:407] Connection header received
0926 16:34:14.558924 (+ 2426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:14.558939 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:14.559499 (+ 560us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:14.559900 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:34:14.561735 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:14.563707 (+ 1972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:14.565614 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:14.566281 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:14.566489 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:14.570730 (+ 4241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:34:14.570744 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:34:14.570766 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:34:14.570819 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:34:14.575158 (+ 4339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:14.576273 (+ 1115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:14.576280 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:14.576286 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:14.576516 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:14.577025 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:14.577029 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:14.577031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:14.577810 (+ 779us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:34:14.578008 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:14.579200 (+ 1192us) server_negotiation.cc:299] Negotiation successful
0926 16:34:14.579606 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":493,"threads_started":1}
I20260926 16:34:14.580825 21683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:14.555294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.2:33857 (local address 127.6.33.1:44743)
0926 16:34:14.555619 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:14.555668 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 16:34:14.556289 (+ 621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:34:14.560011 (+ 3722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:34:14.560019 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:34:14.560633 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:34:14.561473 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:14.561493 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:14.563919 (+ 2426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:14.563923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:14.565457 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:14.565471 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:14.566493 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:14.566500 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:14.566832 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:14.567607 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:34:14.567651 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:34:14.570376 (+ 2725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:34:14.575354 (+ 4978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:34:14.575362 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:34:14.575375 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:34:14.576105 (+ 730us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:34:14.576638 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:34:14.576641 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:34:14.576643 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:34:14.576903 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:34:14.578015 (+ 1112us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:34:14.578020 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:34:14.578538 (+ 518us) client_negotiation.cc:769] Sending connection context
0926 16:34:14.578898 (+ 360us) client_negotiation.cc:240] Negotiation successful
0926 16:34:14.579659 (+ 761us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":446,"spinlock_wait_cycles":32256}
W20260926 16:34:14.583688 21304 leader_election.cc:341] T e57c32124a154d2bbfccf97dcf62dcfa P fa9aad2f0e384189872e794e35bbad29 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ca053d21d9304c0aac25ce7dc6fdacf6 (127.6.33.3:45107): Not found: Tablet not found: e57c32124a154d2bbfccf97dcf62dcfa
W20260926 16:34:14.584025 21302 leader_election.cc:341] T e57c32124a154d2bbfccf97dcf62dcfa P fa9aad2f0e384189872e794e35bbad29 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c6ed5f76dd9e4a4593e96a1eaad01553 (127.6.33.2:33857): Not found: Tablet not found: e57c32124a154d2bbfccf97dcf62dcfa
I20260926 16:34:14.584208 21302 leader_election.cc:302] T e57c32124a154d2bbfccf97dcf62dcfa P fa9aad2f0e384189872e794e35bbad29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fa9aad2f0e384189872e794e35bbad29; no voters: c6ed5f76dd9e4a4593e96a1eaad01553, ca053d21d9304c0aac25ce7dc6fdacf6
I20260926 16:34:14.584657 21685 raft_consensus.cc:2726] T e57c32124a154d2bbfccf97dcf62dcfa P fa9aad2f0e384189872e794e35bbad29 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 16:34:14.987826 21689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:14.986892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36283 (local address 127.6.33.3:51691)
0926 16:34:14.987434 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:14.987576 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":289,"threads_started":1}
I20260926 16:34:15.075210 21683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:15.074773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36283 (local address 127.6.33.1:48123)
0926 16:34:15.074947 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:15.075035 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260926 16:34:15.530722 21690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:15.529991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36283 (local address 127.6.33.2:44009)
0926 16:34:15.530452 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:15.530533 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":236,"threads_started":1}
I20260926 16:34:15.989499 21691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:15.988837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36283 (local address 127.6.33.3:44471)
0926 16:34:15.989263 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:15.989344 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":216,"threads_started":1}
I20260926 16:34:16.076869 21692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:16.076127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36283 (local address 127.6.33.1:51883)
0926 16:34:16.076558 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:16.076635 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":215,"threads_started":1}
I20260926 16:34:16.532560 21693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:16.531730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36283 (local address 127.6.33.2:34241)
0926 16:34:16.532281 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:16.532363 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":332,"threads_started":1}
I20260926 16:34:16.759492 21694 raft_consensus.cc:482] T e57c32124a154d2bbfccf97dcf62dcfa P fa9aad2f0e384189872e794e35bbad29 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:34:16.759742 21694 raft_consensus.cc:504] T e57c32124a154d2bbfccf97dcf62dcfa P fa9aad2f0e384189872e794e35bbad29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ed5f76dd9e4a4593e96a1eaad01553" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 33857 } } peers { permanent_uuid: "ca053d21d9304c0aac25ce7dc6fdacf6" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45107 } } peers { permanent_uuid: "fa9aad2f0e384189872e794e35bbad29" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 34095 } }
I20260926 16:34:16.760485 21694 leader_election.cc:288] T e57c32124a154d2bbfccf97dcf62dcfa P fa9aad2f0e384189872e794e35bbad29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c6ed5f76dd9e4a4593e96a1eaad01553 (127.6.33.2:33857), ca053d21d9304c0aac25ce7dc6fdacf6 (127.6.33.3:45107)
W20260926 16:34:16.761696 21302 leader_election.cc:341] T e57c32124a154d2bbfccf97dcf62dcfa P fa9aad2f0e384189872e794e35bbad29 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c6ed5f76dd9e4a4593e96a1eaad01553 (127.6.33.2:33857): Not found: Tablet not found: e57c32124a154d2bbfccf97dcf62dcfa
W20260926 16:34:16.761991 21304 leader_election.cc:341] T e57c32124a154d2bbfccf97dcf62dcfa P fa9aad2f0e384189872e794e35bbad29 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ca053d21d9304c0aac25ce7dc6fdacf6 (127.6.33.3:45107): Not found: Tablet not found: e57c32124a154d2bbfccf97dcf62dcfa
I20260926 16:34:16.762138 21304 leader_election.cc:302] T e57c32124a154d2bbfccf97dcf62dcfa P fa9aad2f0e384189872e794e35bbad29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fa9aad2f0e384189872e794e35bbad29; no voters: c6ed5f76dd9e4a4593e96a1eaad01553, ca053d21d9304c0aac25ce7dc6fdacf6
I20260926 16:34:16.762457 21694 raft_consensus.cc:2726] T e57c32124a154d2bbfccf97dcf62dcfa P fa9aad2f0e384189872e794e35bbad29 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 16:34:16.991074 21695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:16.990366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36283 (local address 127.6.33.3:48051)
0926 16:34:16.990835 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:16.990913 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":269,"threads_started":1}
I20260926 16:34:17.078691 21696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:17.077830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36283 (local address 127.6.33.1:51651)
0926 16:34:17.078369 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:17.078474 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":289,"threads_started":1}
I20260926 16:34:17.534247 21697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:17.533494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36283 (local address 127.6.33.2:58179)
0926 16:34:17.533961 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:17.534042 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":230,"threads_started":1}
I20260926 16:34:17.993007 21698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:17.992036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36283 (local address 127.6.33.3:46319)
0926 16:34:17.992655 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:17.992800 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":330,"threads_started":1}
I20260926 16:34:18.080458 21699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:18.079706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36283 (local address 127.6.33.1:32993)
0926 16:34:18.080182 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:18.080261 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":229,"threads_started":1}
I20260926 16:34:18.536170 21700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:18.535379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36283 (local address 127.6.33.2:55917)
0926 16:34:18.535875 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:18.535962 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":260,"threads_started":1}
I20260926 16:34:18.564332 21701 raft_consensus.cc:482] T e57c32124a154d2bbfccf97dcf62dcfa P fa9aad2f0e384189872e794e35bbad29 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:34:18.564530 21701 raft_consensus.cc:504] T e57c32124a154d2bbfccf97dcf62dcfa P fa9aad2f0e384189872e794e35bbad29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ed5f76dd9e4a4593e96a1eaad01553" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 33857 } } peers { permanent_uuid: "ca053d21d9304c0aac25ce7dc6fdacf6" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45107 } } peers { permanent_uuid: "fa9aad2f0e384189872e794e35bbad29" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 34095 } }
I20260926 16:34:18.565269 21701 leader_election.cc:288] T e57c32124a154d2bbfccf97dcf62dcfa P fa9aad2f0e384189872e794e35bbad29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c6ed5f76dd9e4a4593e96a1eaad01553 (127.6.33.2:33857), ca053d21d9304c0aac25ce7dc6fdacf6 (127.6.33.3:45107)
W20260926 16:34:18.566164 21302 leader_election.cc:341] T e57c32124a154d2bbfccf97dcf62dcfa P fa9aad2f0e384189872e794e35bbad29 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c6ed5f76dd9e4a4593e96a1eaad01553 (127.6.33.2:33857): Not found: Tablet not found: e57c32124a154d2bbfccf97dcf62dcfa
W20260926 16:34:18.566349 21304 leader_election.cc:341] T e57c32124a154d2bbfccf97dcf62dcfa P fa9aad2f0e384189872e794e35bbad29 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ca053d21d9304c0aac25ce7dc6fdacf6 (127.6.33.3:45107): Not found: Tablet not found: e57c32124a154d2bbfccf97dcf62dcfa
I20260926 16:34:18.566516 21304 leader_election.cc:302] T e57c32124a154d2bbfccf97dcf62dcfa P fa9aad2f0e384189872e794e35bbad29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fa9aad2f0e384189872e794e35bbad29; no voters: c6ed5f76dd9e4a4593e96a1eaad01553, ca053d21d9304c0aac25ce7dc6fdacf6
I20260926 16:34:18.566781 21701 raft_consensus.cc:2726] T e57c32124a154d2bbfccf97dcf62dcfa P fa9aad2f0e384189872e794e35bbad29 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 16:34:18.994714 21702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:18.994017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36283 (local address 127.6.33.3:57263)
0926 16:34:18.994440 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:18.994532 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":227,"threads_started":1}
I20260926 16:34:19.082293 21703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:19.081520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36283 (local address 127.6.33.1:57457)
0926 16:34:19.082025 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:19.082105 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":235,"threads_started":1}
I20260926 16:34:19.538095 21704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:19.537263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36283 (local address 127.6.33.2:58179)
0926 16:34:19.537739 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:19.537830 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":248,"threads_started":1}
I20260926 16:34:19.996533 21705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:19.995703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36283 (local address 127.6.33.3:60537)
0926 16:34:19.996239 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:19.996334 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":252,"threads_started":1}
I20260926 16:34:20.084131 21706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:20.083306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36283 (local address 127.6.33.1:38109)
0926 16:34:20.083829 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:20.083922 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":277,"threads_started":1}
I20260926 16:34:20.518543 21707 raft_consensus.cc:482] T e57c32124a154d2bbfccf97dcf62dcfa P fa9aad2f0e384189872e794e35bbad29 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:34:20.518824 21707 raft_consensus.cc:504] T e57c32124a154d2bbfccf97dcf62dcfa P fa9aad2f0e384189872e794e35bbad29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ed5f76dd9e4a4593e96a1eaad01553" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 33857 } } peers { permanent_uuid: "ca053d21d9304c0aac25ce7dc6fdacf6" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45107 } } peers { permanent_uuid: "fa9aad2f0e384189872e794e35bbad29" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 34095 } }
I20260926 16:34:20.519600 21707 leader_election.cc:288] T e57c32124a154d2bbfccf97dcf62dcfa P fa9aad2f0e384189872e794e35bbad29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c6ed5f76dd9e4a4593e96a1eaad01553 (127.6.33.2:33857), ca053d21d9304c0aac25ce7dc6fdacf6 (127.6.33.3:45107)
W20260926 16:34:20.520578 21302 leader_election.cc:341] T e57c32124a154d2bbfccf97dcf62dcfa P fa9aad2f0e384189872e794e35bbad29 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c6ed5f76dd9e4a4593e96a1eaad01553 (127.6.33.2:33857): Not found: Tablet not found: e57c32124a154d2bbfccf97dcf62dcfa
W20260926 16:34:20.520802 21304 leader_election.cc:341] T e57c32124a154d2bbfccf97dcf62dcfa P fa9aad2f0e384189872e794e35bbad29 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ca053d21d9304c0aac25ce7dc6fdacf6 (127.6.33.3:45107): Not found: Tablet not found: e57c32124a154d2bbfccf97dcf62dcfa
I20260926 16:34:20.520967 21304 leader_election.cc:302] T e57c32124a154d2bbfccf97dcf62dcfa P fa9aad2f0e384189872e794e35bbad29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fa9aad2f0e384189872e794e35bbad29; no voters: c6ed5f76dd9e4a4593e96a1eaad01553, ca053d21d9304c0aac25ce7dc6fdacf6
I20260926 16:34:20.521227 21707 raft_consensus.cc:2726] T e57c32124a154d2bbfccf97dcf62dcfa P fa9aad2f0e384189872e794e35bbad29 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 16:34:20.539997 21708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:20.539212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36283 (local address 127.6.33.2:59973)
0926 16:34:20.539726 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:20.539806 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":226,"threads_started":1}
I20260926 16:34:20.998512 21709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:20.997610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36283 (local address 127.6.33.3:33305)
0926 16:34:20.998203 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:20.998301 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":291,"threads_started":1}
I20260926 16:34:21.085805 21710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:21.085111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36283 (local address 127.6.33.1:37699)
0926 16:34:21.085556 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:21.085640 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":221,"threads_started":1}
I20260926 16:34:21.542004 21711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:21.541246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36283 (local address 127.6.33.2:56317)
0926 16:34:21.541745 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:21.541829 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":254,"threads_started":1}
I20260926 16:34:22.000691 21712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:21.999660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36283 (local address 127.6.33.3:35129)
0926 16:34:22.000266 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:22.000381 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":260,"threads_started":1}
I20260926 16:34:22.088035 21713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:22.086977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36283 (local address 127.6.33.1:48161)
0926 16:34:22.087665 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:22.087774 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":364,"threads_started":1}
I20260926 16:34:22.543949 21714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:22.543120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36283 (local address 127.6.33.2:37103)
0926 16:34:22.543665 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:22.543770 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":251,"threads_started":1}
I20260926 16:34:23.002609 21715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:23.001862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36283 (local address 127.6.33.3:44357)
0926 16:34:23.002339 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:23.002421 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":244,"threads_started":1}
I20260926 16:34:23.090405 21716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:23.089459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36283 (local address 127.6.33.1:54685)
0926 16:34:23.090016 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:23.090143 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":265,"threads_started":1}
I20260926 16:34:23.546061 21717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:23.545127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36283 (local address 127.6.33.2:53583)
0926 16:34:23.545743 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:23.545829 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":354,"threads_started":1}
I20260926 16:34:24.004292 21718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:24.003534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36283 (local address 127.6.33.3:46403)
0926 16:34:24.003966 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:24.004079 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":227,"threads_started":1}
I20260926 16:34:24.092133 21719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:24.091414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36283 (local address 127.6.33.1:37711)
0926 16:34:24.091866 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:24.091949 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":240,"threads_started":1}
I20260926 16:34:24.547873 21720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:24.547051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36283 (local address 127.6.33.2:37841)
0926 16:34:24.547605 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:24.547699 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":251,"threads_started":1}
I20260926 16:34:25.006222 21721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:25.005301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36283 (local address 127.6.33.3:36531)
0926 16:34:25.005906 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:25.006042 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":333,"threads_started":1}
I20260926 16:34:25.094053 21722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:25.093239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36283 (local address 127.6.33.1:53295)
0926 16:34:25.093788 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:25.093880 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":301,"threads_started":1}
I20260926 16:34:25.549938 21723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:25.549063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36283 (local address 127.6.33.2:33175)
0926 16:34:25.549661 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:25.549744 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":307,"threads_started":1}
I20260926 16:34:26.008045 21724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:26.007264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36283 (local address 127.6.33.3:53131)
0926 16:34:26.007766 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:26.007863 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":270,"threads_started":1}
I20260926 16:34:26.095773 21725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:26.094901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36283 (local address 127.6.33.1:48531)
0926 16:34:26.095387 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:26.095551 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":250,"threads_started":1}
I20260926 16:34:26.551509 21726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:26.550842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36283 (local address 127.6.33.2:60569)
0926 16:34:26.551271 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:26.551348 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":210,"threads_started":1}
I20260926 16:34:27.009773 21727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:27.009052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36283 (local address 127.6.33.3:36199)
0926 16:34:27.009533 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:27.009613 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":226,"threads_started":1}
I20260926 16:34:27.097532 21728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:27.096703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36283 (local address 127.6.33.1:44889)
0926 16:34:27.097228 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:27.097313 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":290,"threads_started":1}
I20260926 16:34:27.553341 21729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:27.552429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36283 (local address 127.6.33.2:35451)
0926 16:34:27.553028 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:27.553129 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":258,"threads_started":1}
I20260926 16:34:27.633684 21730 raft_consensus.cc:482] T e57c32124a154d2bbfccf97dcf62dcfa P fa9aad2f0e384189872e794e35bbad29 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:34:27.633908 21730 raft_consensus.cc:504] T e57c32124a154d2bbfccf97dcf62dcfa P fa9aad2f0e384189872e794e35bbad29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ed5f76dd9e4a4593e96a1eaad01553" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 33857 } } peers { permanent_uuid: "ca053d21d9304c0aac25ce7dc6fdacf6" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45107 } } peers { permanent_uuid: "fa9aad2f0e384189872e794e35bbad29" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 34095 } }
I20260926 16:34:27.634574 21730 leader_election.cc:288] T e57c32124a154d2bbfccf97dcf62dcfa P fa9aad2f0e384189872e794e35bbad29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c6ed5f76dd9e4a4593e96a1eaad01553 (127.6.33.2:33857), ca053d21d9304c0aac25ce7dc6fdacf6 (127.6.33.3:45107)
W20260926 16:34:27.635469 21304 leader_election.cc:341] T e57c32124a154d2bbfccf97dcf62dcfa P fa9aad2f0e384189872e794e35bbad29 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ca053d21d9304c0aac25ce7dc6fdacf6 (127.6.33.3:45107): Not found: Tablet not found: e57c32124a154d2bbfccf97dcf62dcfa
W20260926 16:34:27.635637 21302 leader_election.cc:341] T e57c32124a154d2bbfccf97dcf62dcfa P fa9aad2f0e384189872e794e35bbad29 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c6ed5f76dd9e4a4593e96a1eaad01553 (127.6.33.2:33857): Not found: Tablet not found: e57c32124a154d2bbfccf97dcf62dcfa
I20260926 16:34:27.635794 21302 leader_election.cc:302] T e57c32124a154d2bbfccf97dcf62dcfa P fa9aad2f0e384189872e794e35bbad29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fa9aad2f0e384189872e794e35bbad29; no voters: c6ed5f76dd9e4a4593e96a1eaad01553, ca053d21d9304c0aac25ce7dc6fdacf6
I20260926 16:34:27.636068 21730 raft_consensus.cc:2726] T e57c32124a154d2bbfccf97dcf62dcfa P fa9aad2f0e384189872e794e35bbad29 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 16:34:28.011471 21731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:28.010715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36283 (local address 127.6.33.3:34771)
0926 16:34:28.011185 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:28.011272 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":222,"threads_started":1}
I20260926 16:34:28.099699 21732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:28.098806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36283 (local address 127.6.33.1:49303)
0926 16:34:28.099333 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:28.099459 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":275,"threads_started":1}
I20260926 16:34:28.555312 21733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:28.554582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36283 (local address 127.6.33.2:60935)
0926 16:34:28.555060 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:28.555149 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":228,"threads_started":1}
I20260926 16:34:28.979502 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 21252
I20260926 16:34:29.002790 21734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:28.980589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49120 (local address 127.6.33.1:34095)
0926 16:34:28.981149 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:28.981156 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:28.981411 (+ 255us) server_negotiation.cc:407] Connection header received
0926 16:34:28.981610 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:28.981619 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:28.981950 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:28.982145 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:34:28.984007 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:28.985100 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:28.986507 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:28.987054 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:28.987151 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:28.996976 (+ 9825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:34:28.996993 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:34:28.997009 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:34:28.997052 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:34:28.999885 (+ 2833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:29.000653 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:29.000660 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:29.000662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:29.000845 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:29.001189 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:29.001192 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:29.001194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:29.001554 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:34:29.001771 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:29.002194 (+ 423us) server_negotiation.cc:299] Negotiation successful
0926 16:34:29.002399 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":280,"threads_started":1}
I20260926 16:34:29.004105 21412 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:34:29.013793 21736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:29.012926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36283 (local address 127.6.33.3:48121)
0926 16:34:29.013527 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:29.013632 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":283,"threads_started":1}
I20260926 16:34:29.095798 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 21252
I20260926 16:34:29.115512 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 21420
I20260926 16:34:29.130280 21737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:29.116315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45424 (local address 127.6.33.2:33857)
0926 16:34:29.116877 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:29.116883 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:29.116907 (+ 24us) server_negotiation.cc:407] Connection header received
0926 16:34:29.116966 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:29.116969 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:29.117030 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:29.117122 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:34:29.118632 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:29.119510 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:29.121055 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:29.121482 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:29.121585 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:29.124479 (+ 2894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:34:29.124486 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:34:29.124490 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:34:29.124527 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:34:29.127583 (+ 3056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:29.128284 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:29.128289 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:29.128291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:29.128441 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:29.128922 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:29.128926 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:29.128929 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:29.129277 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:34:29.129503 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:29.129733 (+ 230us) server_negotiation.cc:299] Negotiation successful
0926 16:34:29.129958 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":281,"threads_started":1}
I20260926 16:34:29.131079 21543 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:34:29.216254 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 21420
I20260926 16:34:29.234921 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 21551
I20260926 16:34:29.248627 21739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:29.235746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49134 (local address 127.6.33.3:45107)
0926 16:34:29.236338 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:29.236343 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:29.236361 (+ 18us) server_negotiation.cc:407] Connection header received
0926 16:34:29.236421 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:29.236427 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:29.236502 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:29.236601 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:34:29.238052 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:29.238916 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:29.240167 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:29.240631 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:29.240735 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:29.243456 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:34:29.243461 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:34:29.243465 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:34:29.243496 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:34:29.246196 (+ 2700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:29.246885 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:29.246889 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:29.246891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:29.247046 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:29.247370 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:29.247373 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:29.247375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:29.247626 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:34:29.247880 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:29.248033 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 16:34:29.248202 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":322,"threads_started":1}
I20260926 16:34:29.249306 21674 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:34:29.333302 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 21551
2026-09-26T16:34:29Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24434 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:34:29 dist-test-slave-khd3 krb5kdc[21743](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:34:29 dist-test-slave-khd3 krb5kdc[21743](info): set up 2 sockets
Sep 26 16:34:29 dist-test-slave-khd3 krb5kdc[21743](info): commencing operation
krb5kdc: starting...
W20260926 16:34:30.466547 6276 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Sep 26 16:34:30 dist-test-slave-khd3 krb5kdc[21743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:34:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:34:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab.
Entry for principal kudu/127.6.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab.
I20260926 16:34:30.718302 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:38429
--webserver_interface=127.6.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:39917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.33.62:38429
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:34:31.052780 21756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:34:31.053102 21756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:34:31.053193 21756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:34:31.060603 21756 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 16:34:31.060724 21756 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 16:34:31.060781 21756 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 16:34:31.060902 21756 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:34:31.060959 21756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:34:31.061028 21756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:34:31.061074 21756 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:34:31.061126 21756 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:34:31.071568 21756 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:39917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.6.33.62:38429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:38429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:34:31.072933 21756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:34:31.074229 21756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:34:31.086885 21762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:34:31.086905 21764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:34:31.086990 21761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:34:31.087533 21756 server_base.cc:1034] running on GCE node
I20260926 16:34:31.088500 21756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:34:31.089970 21756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:34:31.091189 21756 hybrid_clock.cc:648] HybridClock initialized: now 1790440471091101 us; error 60 us; skew 500 ppm
I20260926 16:34:31.091509 21756 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:34:31 dist-test-slave-khd3 krb5kdc[21743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440471, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:34:31.095098 21756 init.cc:377] Logged in from keytab as kudu/127.6.33.62@KRBTEST.COM (short username kudu)
I20260926 16:34:31.096568 21756 webserver.cc:460] Webserver started at http://127.6.33.62:36315/ using document root <none> and password file <none>
I20260926 16:34:31.097301 21756 fs_manager.cc:359] Metadata directory not provided
I20260926 16:34:31.097388 21756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:34:31.097647 21756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:34:31.100625 21756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/master-0/data/instance:
uuid: "5856adf7a90443d69e7c54ca18a824a3"
format_stamp: "Formatted at 2026-09-26 16:34:31 on dist-test-slave-khd3"
server_key: "592738d743631c0b10b072f0afdfd34d"
server_key_iv: "a103f64f7d5298cbe1fee8d4a4d0b3a9"
server_key_version: "encryptionkey@0"
I20260926 16:34:31.101500 21756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/master-0/wal/instance:
uuid: "5856adf7a90443d69e7c54ca18a824a3"
format_stamp: "Formatted at 2026-09-26 16:34:31 on dist-test-slave-khd3"
server_key: "592738d743631c0b10b072f0afdfd34d"
server_key_iv: "a103f64f7d5298cbe1fee8d4a4d0b3a9"
server_key_version: "encryptionkey@0"
I20260926 16:34:31.106729 21756 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 16:34:31.110204 21771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:34:31.111613 21756 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:34:31.111795 21756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/master-0/wal
uuid: "5856adf7a90443d69e7c54ca18a824a3"
format_stamp: "Formatted at 2026-09-26 16:34:31 on dist-test-slave-khd3"
server_key: "592738d743631c0b10b072f0afdfd34d"
server_key_iv: "a103f64f7d5298cbe1fee8d4a4d0b3a9"
server_key_version: "encryptionkey@0"
I20260926 16:34:31.112013 21756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:34:31.131573 21756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:34:31.135843 21756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:34:31.136142 21756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:34:31.155576 21756 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.62:38429
I20260926 16:34:31.155609 21822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.62:38429 every 8 connection(s)
I20260926 16:34:31.157087 21756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
I20260926 16:34:31.161075 21823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:34:31.161793 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 21756
I20260926 16:34:31.162027 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/master-0/wal/instance
I20260926 16:34:31.162706 6276 external_mini_cluster.cc:1428] Setting key 730d12fd694936213a9a58da85f5f967
I20260926 16:34:31.175290 21823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3: Bootstrap starting.
Sep 26 16:34:31 dist-test-slave-khd3 krb5kdc[21743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:34:31.179049 21823 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3: Neither blocks nor log segments found. Creating new log.
I20260926 16:34:31.180270 21823 log.cc:824] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3: Log is configured to *not* fsync() on all Append() calls
I20260926 16:34:31.183279 21823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3: No bootstrap required, opened a new log
I20260926 16:34:31.186664 21826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:31.164708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58440 (local address 127.6.33.62:38429)
0926 16:34:31.165582 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:31.165596 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:31.165639 (+ 43us) server_negotiation.cc:407] Connection header received
0926 16:34:31.167364 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:31.167391 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:31.167807 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:31.168158 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:34:31.170018 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:31.171316 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:31.173306 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:31.174071 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:31.174194 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:31.178941 (+ 4747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:34:31.178952 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:34:31.178966 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:34:31.179002 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:34:31.182527 (+ 3525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:31.183216 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:31.183222 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:31.183229 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:31.183459 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:31.183853 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:31.183858 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:31.183863 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:31.184377 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:34:31.184683 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:31.185178 (+ 495us) server_negotiation.cc:299] Negotiation successful
0926 16:34:31.185640 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":395,"threads_started":1}
I20260926 16:34:31.191203 21823 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5856adf7a90443d69e7c54ca18a824a3" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 38429 } }
I20260926 16:34:31.191581 21823 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:34:31.191704 21823 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5856adf7a90443d69e7c54ca18a824a3, State: Initialized, Role: FOLLOWER
I20260926 16:34:31.192193 21823 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5856adf7a90443d69e7c54ca18a824a3" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 38429 } }
I20260926 16:34:31.192396 21823 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:34:31.192490 21823 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:34:31.192611 21823 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:34:31.195122 21823 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5856adf7a90443d69e7c54ca18a824a3" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 38429 } }
I20260926 16:34:31.195595 21823 leader_election.cc:302] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5856adf7a90443d69e7c54ca18a824a3; no voters:
I20260926 16:34:31.196136 21823 leader_election.cc:288] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:34:31.196300 21828 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:34:31.197088 21828 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3 [term 1 LEADER]: Becoming Leader. State: Replica: 5856adf7a90443d69e7c54ca18a824a3, State: Running, Role: LEADER
I20260926 16:34:31.197576 21828 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5856adf7a90443d69e7c54ca18a824a3" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 38429 } }
I20260926 16:34:31.198431 21823 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:34:31.200699 21830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5856adf7a90443d69e7c54ca18a824a3. Latest consensus state: current_term: 1 leader_uuid: "5856adf7a90443d69e7c54ca18a824a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5856adf7a90443d69e7c54ca18a824a3" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 38429 } } }
I20260926 16:34:31.200974 21829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5856adf7a90443d69e7c54ca18a824a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5856adf7a90443d69e7c54ca18a824a3" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 38429 } } }
I20260926 16:34:31.201208 21830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3 [sys.catalog]: This master's current role is: LEADER
I20260926 16:34:31.201272 21829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3 [sys.catalog]: This master's current role is: LEADER
I20260926 16:34:31.206169 21834 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:34:31.212285 21834 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:34:31.221966 21834 catalog_manager.cc:1347] Generated new cluster ID: dcb4b0b790d34c829828b3ff2f70dc6e
I20260926 16:34:31.222098 21834 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:34:31.252367 21834 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:34:31.254482 21834 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:34:31.261724 21834 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3: Generated new TSK 0
I20260926 16:34:31.262810 21834 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.33.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab.
Entry for principal kudu/127.6.33.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab.
I20260926 16:34:31.324853 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.1:0
--local_ip_for_outbound_sockets=127.6.33.1
--webserver_interface=127.6.33.1
--webserver_port=0
--tserver_master_addrs=127.6.33.62:38429
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab
--principal=kudu/127.6.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:39917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:34:31.662518 21849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:34:31.662878 21849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:34:31.663056 21849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:34:31.671203 21849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:34:31.671370 21849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:34:31.671543 21849 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.1
I20260926 16:34:31.682452 21849 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:39917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab
--principal=kudu/127.6.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.33.1
--webserver_port=0
--tserver_master_addrs=127.6.33.62:38429
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.1
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:34:31.683894 21849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:34:31.685355 21849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:34:31.699723 21855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:34:31.699671 21857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:34:31.699641 21854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:34:31.700456 21849 server_base.cc:1034] running on GCE node
I20260926 16:34:31.701159 21849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:34:31.702205 21849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:34:31.703423 21849 hybrid_clock.cc:648] HybridClock initialized: now 1790440471703373 us; error 46 us; skew 500 ppm
I20260926 16:34:31.703783 21849 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:34:31 dist-test-slave-khd3 krb5kdc[21743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440471, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:34:31.707291 21849 init.cc:377] Logged in from keytab as kudu/127.6.33.1@KRBTEST.COM (short username kudu)
I20260926 16:34:31.709000 21849 webserver.cc:460] Webserver started at http://127.6.33.1:41389/ using document root <none> and password file <none>
I20260926 16:34:31.709726 21849 fs_manager.cc:359] Metadata directory not provided
I20260926 16:34:31.709882 21849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:34:31.710220 21849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:34:31.713351 21849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-0/data/instance:
uuid: "f1f4f18f75344a569ccd2bf27a10f8d7"
format_stamp: "Formatted at 2026-09-26 16:34:31 on dist-test-slave-khd3"
server_key: "4389302381262e875ef68fd0d70de23a"
server_key_iv: "8f43022fb015dd8520e5a24fa782ed6f"
server_key_version: "encryptionkey@0"
I20260926 16:34:31.714116 21849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-0/wal/instance:
uuid: "f1f4f18f75344a569ccd2bf27a10f8d7"
format_stamp: "Formatted at 2026-09-26 16:34:31 on dist-test-slave-khd3"
server_key: "4389302381262e875ef68fd0d70de23a"
server_key_iv: "8f43022fb015dd8520e5a24fa782ed6f"
server_key_version: "encryptionkey@0"
I20260926 16:34:31.719533 21849 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 16:34:31.723968 21864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:34:31.725628 21849 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 16:34:31.725811 21849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-0/wal
uuid: "f1f4f18f75344a569ccd2bf27a10f8d7"
format_stamp: "Formatted at 2026-09-26 16:34:31 on dist-test-slave-khd3"
server_key: "4389302381262e875ef68fd0d70de23a"
server_key_iv: "8f43022fb015dd8520e5a24fa782ed6f"
server_key_version: "encryptionkey@0"
I20260926 16:34:31.726049 21849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:34:31.749090 21849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:34:31.753623 21849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:34:31.753979 21849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:34:31.755008 21849 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:34:31.756314 21849 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:34:31.756395 21849 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:34:31.756547 21849 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:34:31.756613 21849 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:34:31.790505 21849 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.1:41569
I20260926 16:34:31.790555 21976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.1:41569 every 8 connection(s)
I20260926 16:34:31.791922 21849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
I20260926 16:34:31.801831 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 21849
I20260926 16:34:31.802052 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-0/wal/instance
I20260926 16:34:31.802807 6276 external_mini_cluster.cc:1428] Setting key 69a31a09ab0c04ad74dca5fafd27c810
Sep 26 16:34:31 dist-test-slave-khd3 krb5kdc[21743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440471, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.1@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:34:31.813611 21979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:31.794654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:45799 (local address 127.6.33.62:38429)
0926 16:34:31.795102 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:31.795107 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:31.796088 (+ 981us) server_negotiation.cc:407] Connection header received
0926 16:34:31.798019 (+ 1931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:31.798025 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:31.798115 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:31.798236 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:34:31.799732 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:31.800446 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:31.802010 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:31.802482 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:31.802631 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:31.806639 (+ 4008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:34:31.806648 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:34:31.806656 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:34:31.806701 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:34:31.810369 (+ 3668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:31.811148 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:31.811156 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:31.811162 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:31.811324 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:31.811746 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:31.811753 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:31.811759 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:31.812142 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:34:31.812318 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:31.813061 (+ 743us) server_negotiation.cc:299] Negotiation successful
0926 16:34:31.813347 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":223,"threads_started":1}
I20260926 16:34:31.814940 21980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:31.795232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:38429 (local address 127.6.33.1:45799)
0926 16:34:31.795924 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:31.795963 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 16:34:31.797738 (+ 1775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:34:31.798491 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:34:31.798499 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:34:31.798994 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:34:31.799571 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:31.799587 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:31.800608 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:31.800612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:31.801863 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:31.801875 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:31.802703 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:31.802711 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:31.803035 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:31.803836 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:34:31.803869 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:34:31.806408 (+ 2539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:34:31.810538 (+ 4130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:34:31.810545 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:34:31.810559 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:34:31.811009 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:34:31.811443 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:34:31.811446 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:34:31.811448 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:34:31.811626 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:34:31.812269 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:34:31.812274 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:34:31.812616 (+ 342us) client_negotiation.cc:769] Sending connection context
0926 16:34:31.812903 (+ 287us) client_negotiation.cc:240] Negotiation successful
0926 16:34:31.813394 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":261,"threads_started":1}
I20260926 16:34:31.816658 21977 heartbeater.cc:347] Connected to a master server at 127.6.33.62:38429
I20260926 16:34:31.817056 21977 heartbeater.cc:464] Registering TS with master...
I20260926 16:34:31.817798 21977 heartbeater.cc:511] Master 127.6.33.62:38429 requested a full tablet report, sending...
I20260926 16:34:31.819859 21788 ts_manager.cc:194] Registered new tserver with Master: f1f4f18f75344a569ccd2bf27a10f8d7 (127.6.33.1:41569)
I20260926 16:34:31.821595 21788 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.33.1@KRBTEST.COM'} at 127.6.33.1:45799
WARNING: no policy specified for kudu/127.6.33.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab.
Entry for principal kudu/127.6.33.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab.
I20260926 16:34:31.856737 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.2:0
--local_ip_for_outbound_sockets=127.6.33.2
--webserver_interface=127.6.33.2
--webserver_port=0
--tserver_master_addrs=127.6.33.62:38429
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab
--principal=kudu/127.6.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:39917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:34:32.190302 21983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:34:32.190666 21983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:34:32.190845 21983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:34:32.198526 21983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:34:32.198673 21983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:34:32.198849 21983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.2
I20260926 16:34:32.209123 21983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:39917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab
--principal=kudu/127.6.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.33.2
--webserver_port=0
--tserver_master_addrs=127.6.33.62:38429
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.2
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:34:32.210666 21983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:34:32.212046 21983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:34:32.225903 21989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:34:32.225981 21988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:34:32.226305 21991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:34:32.227221 21983 server_base.cc:1034] running on GCE node
I20260926 16:34:32.227905 21983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:34:32.229038 21983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:34:32.230237 21983 hybrid_clock.cc:648] HybridClock initialized: now 1790440472230177 us; error 48 us; skew 500 ppm
I20260926 16:34:32.230616 21983 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:34:32 dist-test-slave-khd3 krb5kdc[21743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440472, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:34:32.233946 21983 init.cc:377] Logged in from keytab as kudu/127.6.33.2@KRBTEST.COM (short username kudu)
I20260926 16:34:32.235544 21983 webserver.cc:460] Webserver started at http://127.6.33.2:38901/ using document root <none> and password file <none>
I20260926 16:34:32.236228 21983 fs_manager.cc:359] Metadata directory not provided
I20260926 16:34:32.236341 21983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:34:32.236598 21983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:34:32.239807 21983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-1/data/instance:
uuid: "fa5eb6546b9642e5b64b22da3a5298ca"
format_stamp: "Formatted at 2026-09-26 16:34:32 on dist-test-slave-khd3"
server_key: "45135926b847a97d0c7f4a5c30a68dd2"
server_key_iv: "f68ea020c7bf848a946f845f2a8e7e12"
server_key_version: "encryptionkey@0"
I20260926 16:34:32.240548 21983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-1/wal/instance:
uuid: "fa5eb6546b9642e5b64b22da3a5298ca"
format_stamp: "Formatted at 2026-09-26 16:34:32 on dist-test-slave-khd3"
server_key: "45135926b847a97d0c7f4a5c30a68dd2"
server_key_iv: "f68ea020c7bf848a946f845f2a8e7e12"
server_key_version: "encryptionkey@0"
I20260926 16:34:32.246042 21983 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 16:34:32.249589 21998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:34:32.250955 21983 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:34:32.251133 21983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-1/wal
uuid: "fa5eb6546b9642e5b64b22da3a5298ca"
format_stamp: "Formatted at 2026-09-26 16:34:32 on dist-test-slave-khd3"
server_key: "45135926b847a97d0c7f4a5c30a68dd2"
server_key_iv: "f68ea020c7bf848a946f845f2a8e7e12"
server_key_version: "encryptionkey@0"
I20260926 16:34:32.251310 21983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:34:32.265340 21983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:34:32.269723 21983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:34:32.270005 21983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:34:32.270936 21983 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:34:32.272271 21983 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:34:32.272348 21983 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:34:32.272514 21983 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:34:32.272571 21983 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:34:32.307327 21983 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.2:32913
I20260926 16:34:32.307374 22110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.2:32913 every 8 connection(s)
I20260926 16:34:32.308593 21983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
I20260926 16:34:32.310356 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 21983
I20260926 16:34:32.310554 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-1/wal/instance
I20260926 16:34:32.311165 6276 external_mini_cluster.cc:1428] Setting key 6f39730c926d8357265560761a8ca7f8
Sep 26 16:34:32 dist-test-slave-khd3 krb5kdc[21743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440472, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.2@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:34:32.330782 21979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:32.311497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:50485 (local address 127.6.33.62:38429)
0926 16:34:32.311685 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:32.311689 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:32.313104 (+ 1415us) server_negotiation.cc:407] Connection header received
0926 16:34:32.314996 (+ 1892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:32.315002 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:32.315085 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:32.315207 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:34:32.316854 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:32.317563 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:32.319100 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:32.319548 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:32.319697 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:32.323472 (+ 3775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:34:32.323481 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:34:32.323488 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:34:32.323535 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:34:32.327299 (+ 3764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:32.328055 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:32.328064 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:32.328070 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:32.328239 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:32.328665 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:32.328672 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:32.328678 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:32.329100 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:34:32.329252 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:32.330024 (+ 772us) server_negotiation.cc:299] Negotiation successful
0926 16:34:32.330314 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 16:34:32.331511 22113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:32.312099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:38429 (local address 127.6.33.2:50485)
0926 16:34:32.312946 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:32.312993 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 16:34:32.314730 (+ 1737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:34:32.315486 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:34:32.315495 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:34:32.316052 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:34:32.316660 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:32.316678 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:32.317719 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:32.317726 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:32.318961 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:32.318972 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:32.319785 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:32.319790 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:32.319997 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:32.320734 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:34:32.320818 (+ 84us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:34:32.323220 (+ 2402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:34:32.327456 (+ 4236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:34:32.327463 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:34:32.327476 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:34:32.327884 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:34:32.328348 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:34:32.328351 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:34:32.328353 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:34:32.328546 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:34:32.329226 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:34:32.329231 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:34:32.329581 (+ 350us) client_negotiation.cc:769] Sending connection context
0926 16:34:32.329854 (+ 273us) client_negotiation.cc:240] Negotiation successful
0926 16:34:32.330339 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":336,"threads_started":1}
I20260926 16:34:32.333114 22111 heartbeater.cc:347] Connected to a master server at 127.6.33.62:38429
I20260926 16:34:32.333456 22111 heartbeater.cc:464] Registering TS with master...
I20260926 16:34:32.334167 22111 heartbeater.cc:511] Master 127.6.33.62:38429 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.33.3@KRBTEST.COM; defaulting to no policy
I20260926 16:34:32.335615 21788 ts_manager.cc:194] Registered new tserver with Master: fa5eb6546b9642e5b64b22da3a5298ca (127.6.33.2:32913)
I20260926 16:34:32.336400 21788 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.33.2@KRBTEST.COM'} at 127.6.33.2:50485
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab.
Entry for principal kudu/127.6.33.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab.
I20260926 16:34:32.368083 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.3:0
--local_ip_for_outbound_sockets=127.6.33.3
--webserver_interface=127.6.33.3
--webserver_port=0
--tserver_master_addrs=127.6.33.62:38429
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab
--principal=kudu/127.6.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:39917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:34:32.694864 22116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:34:32.695163 22116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:34:32.695259 22116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:34:32.702680 22116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:34:32.702809 22116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:34:32.702929 22116 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.3
I20260926 16:34:32.713073 22116 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:39917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab
--principal=kudu/127.6.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.33.3
--webserver_port=0
--tserver_master_addrs=127.6.33.62:38429
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.3
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:34:32.714331 22116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:34:32.715695 22116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:34:32.730072 22121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:34:32.730156 22122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:34:32.730160 22124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:34:32.731662 22116 server_base.cc:1034] running on GCE node
I20260926 16:34:32.732322 22116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:34:32.733347 22116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:34:32.734583 22116 hybrid_clock.cc:648] HybridClock initialized: now 1790440472734540 us; error 39 us; skew 500 ppm
I20260926 16:34:32.734925 22116 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:34:32 dist-test-slave-khd3 krb5kdc[21743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440472, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:34:32.738368 22116 init.cc:377] Logged in from keytab as kudu/127.6.33.3@KRBTEST.COM (short username kudu)
I20260926 16:34:32.739800 22116 webserver.cc:460] Webserver started at http://127.6.33.3:45539/ using document root <none> and password file <none>
I20260926 16:34:32.740502 22116 fs_manager.cc:359] Metadata directory not provided
I20260926 16:34:32.740645 22116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:34:32.741020 22116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:34:32.743968 22116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-2/data/instance:
uuid: "6a190cafd1f94f38ad1c6bf176f8066b"
format_stamp: "Formatted at 2026-09-26 16:34:32 on dist-test-slave-khd3"
server_key: "0beebb995bbc9497045f52e93f40e1f8"
server_key_iv: "d69056b5ed03938b92bee3a940adc843"
server_key_version: "encryptionkey@0"
I20260926 16:34:32.744735 22116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-2/wal/instance:
uuid: "6a190cafd1f94f38ad1c6bf176f8066b"
format_stamp: "Formatted at 2026-09-26 16:34:32 on dist-test-slave-khd3"
server_key: "0beebb995bbc9497045f52e93f40e1f8"
server_key_iv: "d69056b5ed03938b92bee3a940adc843"
server_key_version: "encryptionkey@0"
I20260926 16:34:32.750327 22116 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 16:34:32.753949 22131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:34:32.755445 22116 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:34:32.755625 22116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-2/wal
uuid: "6a190cafd1f94f38ad1c6bf176f8066b"
format_stamp: "Formatted at 2026-09-26 16:34:32 on dist-test-slave-khd3"
server_key: "0beebb995bbc9497045f52e93f40e1f8"
server_key_iv: "d69056b5ed03938b92bee3a940adc843"
server_key_version: "encryptionkey@0"
I20260926 16:34:32.755828 22116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:34:32.779759 22116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:34:32.784490 22116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:34:32.784826 22116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:34:32.785859 22116 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:34:32.787215 22116 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:34:32.787351 22116 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:34:32.787503 22116 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:34:32.787575 22116 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:34:32.820890 22116 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.3:45075
I20260926 16:34:32.820920 22243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.3:45075 every 8 connection(s)
I20260926 16:34:32.822206 22116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
I20260926 16:34:32.824690 21977 heartbeater.cc:503] Master 127.6.33.62:38429 was elected leader, sending a full tablet report...
I20260926 16:34:32.831399 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 22116
I20260926 16:34:32.831620 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-2/wal/instance
I20260926 16:34:32.832198 6276 external_mini_cluster.cc:1428] Setting key 21c491b37196bebd2e7578c3156acbd2
Sep 26 16:34:32 dist-test-slave-khd3 krb5kdc[21743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440472, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.3@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:34:32.843601 21979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:32.824950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.3:41691 (local address 127.6.33.62:38429)
0926 16:34:32.825179 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:32.825185 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:32.826476 (+ 1291us) server_negotiation.cc:407] Connection header received
0926 16:34:32.828342 (+ 1866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:32.828348 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:32.828418 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:32.828512 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:34:32.830086 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:32.831181 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:32.833403 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:32.833841 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:32.833976 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:32.837799 (+ 3823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:34:32.837804 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:34:32.837807 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:34:32.837845 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:34:32.840378 (+ 2533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:32.841219 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:32.841223 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:32.841226 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:32.841363 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:32.841814 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:32.841817 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:32.841820 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:32.842088 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:34:32.842248 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:32.842995 (+ 747us) server_negotiation.cc:299] Negotiation successful
0926 16:34:32.843235 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 16:34:32.844565 22246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:32.825470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:38429 (local address 127.6.33.3:41691)
0926 16:34:32.826301 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:32.826354 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 16:34:32.828075 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:34:32.828708 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:34:32.828716 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:34:32.829304 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:34:32.829911 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:32.829929 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:32.831379 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:32.831383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:32.833234 (+ 1851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:32.833247 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:32.834098 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:32.834109 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:32.834339 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:32.835104 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:34:32.835128 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:34:32.837554 (+ 2426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:34:32.840530 (+ 2976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:34:32.840540 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:34:32.840566 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:34:32.841080 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:34:32.841480 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:34:32.841485 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:34:32.841501 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:34:32.841681 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:34:32.842219 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:34:32.842227 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:34:32.842595 (+ 368us) client_negotiation.cc:769] Sending connection context
0926 16:34:32.842873 (+ 278us) client_negotiation.cc:240] Negotiation successful
0926 16:34:32.843438 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":331,"threads_started":1}
I20260926 16:34:32.846022 22244 heartbeater.cc:347] Connected to a master server at 127.6.33.62:38429
I20260926 16:34:32.846371 22244 heartbeater.cc:464] Registering TS with master...
I20260926 16:34:32.847105 22244 heartbeater.cc:511] Master 127.6.33.62:38429 requested a full tablet report, sending...
I20260926 16:34:32.848487 21788 ts_manager.cc:194] Registered new tserver with Master: 6a190cafd1f94f38ad1c6bf176f8066b (127.6.33.3:45075)
I20260926 16:34:32.849319 21788 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.33.3@KRBTEST.COM'} at 127.6.33.3:41691
I20260926 16:34:32.860399 6276 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:34:32.878610 21979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:32.865252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58450 (local address 127.6.33.62:38429)
0926 16:34:32.865454 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:32.865461 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:32.866055 (+ 594us) server_negotiation.cc:407] Connection header received
0926 16:34:32.866182 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:32.866186 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:32.866252 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:32.866338 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:34:32.867754 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:32.868491 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:32.870076 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:32.870645 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:32.870743 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:32.873349 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:34:32.873354 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:34:32.873358 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:34:32.873391 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:34:32.876023 (+ 2632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:32.876698 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:32.876702 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:32.876704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:32.876882 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:32.877221 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:32.877224 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:32.877226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:32.877548 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:34:32.877733 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:32.878157 (+ 424us) server_negotiation.cc:299] Negotiation successful
0926 16:34:32.878322 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 16:34:32.882771 21788 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58450:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:34:32.889842 21788 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:34:32.924275 22259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:32.903954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56720 (local address 127.6.33.1:41569)
0926 16:34:32.915879 (+ 11925us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:32.915891 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:32.915927 (+ 36us) server_negotiation.cc:407] Connection header received
0926 16:34:32.916042 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:32.916048 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:32.916314 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:32.917356 (+ 1042us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:34:32.917570 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:32.919098 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:32.921333 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:32.922838 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:32.922980 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:32.923226 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:32.923792 (+ 566us) server_negotiation.cc:299] Negotiation successful
0926 16:34:32.923951 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11718,"thread_start_us":11399,"threads_started":1}
I20260926 16:34:32.924353 22258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:32.903486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:41569 (local address 127.0.0.1:56720)
0926 16:34:32.913932 (+ 10446us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:32.913959 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 16:34:32.914102 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:34:32.916603 (+ 2501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:34:32.916608 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:34:32.916636 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:34:32.917092 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:32.917105 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:32.919253 (+ 2148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:32.919260 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:32.921187 (+ 1927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:32.921199 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:32.923369 (+ 2170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:32.923373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:32.923520 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:32.923581 (+ 61us) client_negotiation.cc:769] Sending connection context
0926 16:34:32.923710 (+ 129us) client_negotiation.cc:240] Negotiation successful
0926 16:34:32.923915 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10265,"mutex_wait_us":194,"thread_start_us":10640,"threads_started":1}
I20260926 16:34:32.924567 22257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:32.902818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38010 (local address 127.6.33.2:32913)
0926 16:34:32.915081 (+ 12263us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:32.915091 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:32.915121 (+ 30us) server_negotiation.cc:407] Connection header received
0926 16:34:32.915237 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:32.915259 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:32.915560 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:32.916355 (+ 795us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:34:32.916666 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:32.918378 (+ 1712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:32.921506 (+ 3128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:32.922916 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:32.923039 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:32.923194 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:32.923864 (+ 670us) server_negotiation.cc:299] Negotiation successful
0926 16:34:32.924049 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12077,"thread_start_us":12820,"threads_started":1}
I20260926 16:34:32.925155 22256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:32.902613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.2:32913 (local address 127.0.0.1:38010)
0926 16:34:32.908395 (+ 5782us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:32.908448 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 16:34:32.908704 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:34:32.915829 (+ 7125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:34:32.915840 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:34:32.915884 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:34:32.916404 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:32.916422 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:32.918537 (+ 2115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:32.918543 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:32.921353 (+ 2810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:32.921367 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:32.923249 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:32.923255 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:32.923476 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:32.923577 (+ 101us) client_negotiation.cc:769] Sending connection context
0926 16:34:32.923700 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 16:34:32.924793 (+ 1093us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":899,"mutex_wait_us":1036,"thread_start_us":718,"threads_started":1}
I20260926 16:34:32.925172 22260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:32.904135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.3:45075 (local address 127.0.0.1:37464)
0926 16:34:32.916365 (+ 12230us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:32.916388 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 16:34:32.916509 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:34:32.917084 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:34:32.917088 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:34:32.917112 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:34:32.917505 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:32.917515 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:32.919345 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:32.919351 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:32.921638 (+ 2287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:32.921648 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:32.924407 (+ 2759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:32.924414 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:32.924617 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:32.924654 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 16:34:32.924742 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 16:34:32.924873 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12058,"mutex_wait_us":61,"thread_start_us":11857,"threads_started":1}
I20260926 16:34:32.925755 22261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:32.904662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37464 (local address 127.6.33.3:45075)
0926 16:34:32.915609 (+ 10947us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:32.915619 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:32.916585 (+ 966us) server_negotiation.cc:407] Connection header received
0926 16:34:32.916681 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:32.916686 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:32.916907 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:32.917092 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:34:32.917677 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:32.919183 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:32.921772 (+ 2589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:32.923923 (+ 2151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:32.924040 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:32.924250 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:32.925276 (+ 1026us) server_negotiation.cc:299] Negotiation successful
0926 16:34:32.925460 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10766,"thread_start_us":10617,"threads_started":1}
I20260926 16:34:32.929244 21912 tablet_service.cc:1467] Processing CreateTablet for tablet 57bce21fbf7d4a13aea368cd77d33e2f (DEFAULT_TABLE table=test-table [id=ea06c49cf1b54c9398d1d755a84e58da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:34:32.929251 22179 tablet_service.cc:1467] Processing CreateTablet for tablet 57bce21fbf7d4a13aea368cd77d33e2f (DEFAULT_TABLE table=test-table [id=ea06c49cf1b54c9398d1d755a84e58da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:34:32.930609 22179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57bce21fbf7d4a13aea368cd77d33e2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:34:32.931813 22046 tablet_service.cc:1467] Processing CreateTablet for tablet 57bce21fbf7d4a13aea368cd77d33e2f (DEFAULT_TABLE table=test-table [id=ea06c49cf1b54c9398d1d755a84e58da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:34:32.933056 22046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57bce21fbf7d4a13aea368cd77d33e2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:34:32.933115 21912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57bce21fbf7d4a13aea368cd77d33e2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:34:32.942572 22262 tablet_bootstrap.cc:492] T 57bce21fbf7d4a13aea368cd77d33e2f P 6a190cafd1f94f38ad1c6bf176f8066b: Bootstrap starting.
I20260926 16:34:32.945879 22262 tablet_bootstrap.cc:654] T 57bce21fbf7d4a13aea368cd77d33e2f P 6a190cafd1f94f38ad1c6bf176f8066b: Neither blocks nor log segments found. Creating new log.
I20260926 16:34:32.946955 22262 log.cc:824] T 57bce21fbf7d4a13aea368cd77d33e2f P 6a190cafd1f94f38ad1c6bf176f8066b: Log is configured to *not* fsync() on all Append() calls
I20260926 16:34:32.948132 22263 tablet_bootstrap.cc:492] T 57bce21fbf7d4a13aea368cd77d33e2f P f1f4f18f75344a569ccd2bf27a10f8d7: Bootstrap starting.
I20260926 16:34:32.949127 22264 tablet_bootstrap.cc:492] T 57bce21fbf7d4a13aea368cd77d33e2f P fa5eb6546b9642e5b64b22da3a5298ca: Bootstrap starting.
I20260926 16:34:32.952088 22263 tablet_bootstrap.cc:654] T 57bce21fbf7d4a13aea368cd77d33e2f P f1f4f18f75344a569ccd2bf27a10f8d7: Neither blocks nor log segments found. Creating new log.
I20260926 16:34:32.953194 22263 log.cc:824] T 57bce21fbf7d4a13aea368cd77d33e2f P f1f4f18f75344a569ccd2bf27a10f8d7: Log is configured to *not* fsync() on all Append() calls
I20260926 16:34:32.953841 22264 tablet_bootstrap.cc:654] T 57bce21fbf7d4a13aea368cd77d33e2f P fa5eb6546b9642e5b64b22da3a5298ca: Neither blocks nor log segments found. Creating new log.
I20260926 16:34:32.955128 22264 log.cc:824] T 57bce21fbf7d4a13aea368cd77d33e2f P fa5eb6546b9642e5b64b22da3a5298ca: Log is configured to *not* fsync() on all Append() calls
I20260926 16:34:32.956176 22262 tablet_bootstrap.cc:492] T 57bce21fbf7d4a13aea368cd77d33e2f P 6a190cafd1f94f38ad1c6bf176f8066b: No bootstrap required, opened a new log
I20260926 16:34:32.956176 22263 tablet_bootstrap.cc:492] T 57bce21fbf7d4a13aea368cd77d33e2f P f1f4f18f75344a569ccd2bf27a10f8d7: No bootstrap required, opened a new log
I20260926 16:34:32.956462 22262 ts_tablet_manager.cc:1397] T 57bce21fbf7d4a13aea368cd77d33e2f P 6a190cafd1f94f38ad1c6bf176f8066b: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.008s
I20260926 16:34:32.956462 22263 ts_tablet_manager.cc:1397] T 57bce21fbf7d4a13aea368cd77d33e2f P f1f4f18f75344a569ccd2bf27a10f8d7: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260926 16:34:32.957966 22264 tablet_bootstrap.cc:492] T 57bce21fbf7d4a13aea368cd77d33e2f P fa5eb6546b9642e5b64b22da3a5298ca: No bootstrap required, opened a new log
I20260926 16:34:32.958235 22264 ts_tablet_manager.cc:1397] T 57bce21fbf7d4a13aea368cd77d33e2f P fa5eb6546b9642e5b64b22da3a5298ca: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260926 16:34:32.966454 22264 raft_consensus.cc:348] T 57bce21fbf7d4a13aea368cd77d33e2f P fa5eb6546b9642e5b64b22da3a5298ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5eb6546b9642e5b64b22da3a5298ca" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 32913 } } peers { permanent_uuid: "6a190cafd1f94f38ad1c6bf176f8066b" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45075 } } peers { permanent_uuid: "f1f4f18f75344a569ccd2bf27a10f8d7" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 41569 } }
I20260926 16:34:32.966907 22264 raft_consensus.cc:374] T 57bce21fbf7d4a13aea368cd77d33e2f P fa5eb6546b9642e5b64b22da3a5298ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:34:32.967047 22264 raft_consensus.cc:729] T 57bce21fbf7d4a13aea368cd77d33e2f P fa5eb6546b9642e5b64b22da3a5298ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa5eb6546b9642e5b64b22da3a5298ca, State: Initialized, Role: FOLLOWER
I20260926 16:34:32.967723 22264 consensus_queue.cc:260] T 57bce21fbf7d4a13aea368cd77d33e2f P fa5eb6546b9642e5b64b22da3a5298ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5eb6546b9642e5b64b22da3a5298ca" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 32913 } } peers { permanent_uuid: "6a190cafd1f94f38ad1c6bf176f8066b" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45075 } } peers { permanent_uuid: "f1f4f18f75344a569ccd2bf27a10f8d7" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 41569 } }
I20260926 16:34:32.967685 22263 raft_consensus.cc:348] T 57bce21fbf7d4a13aea368cd77d33e2f P f1f4f18f75344a569ccd2bf27a10f8d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5eb6546b9642e5b64b22da3a5298ca" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 32913 } } peers { permanent_uuid: "6a190cafd1f94f38ad1c6bf176f8066b" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45075 } } peers { permanent_uuid: "f1f4f18f75344a569ccd2bf27a10f8d7" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 41569 } }
I20260926 16:34:32.967686 22262 raft_consensus.cc:348] T 57bce21fbf7d4a13aea368cd77d33e2f P 6a190cafd1f94f38ad1c6bf176f8066b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5eb6546b9642e5b64b22da3a5298ca" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 32913 } } peers { permanent_uuid: "6a190cafd1f94f38ad1c6bf176f8066b" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45075 } } peers { permanent_uuid: "f1f4f18f75344a569ccd2bf27a10f8d7" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 41569 } }
I20260926 16:34:32.968127 22262 raft_consensus.cc:374] T 57bce21fbf7d4a13aea368cd77d33e2f P 6a190cafd1f94f38ad1c6bf176f8066b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:34:32.968127 22263 raft_consensus.cc:374] T 57bce21fbf7d4a13aea368cd77d33e2f P f1f4f18f75344a569ccd2bf27a10f8d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:34:32.968236 22263 raft_consensus.cc:729] T 57bce21fbf7d4a13aea368cd77d33e2f P f1f4f18f75344a569ccd2bf27a10f8d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1f4f18f75344a569ccd2bf27a10f8d7, State: Initialized, Role: FOLLOWER
I20260926 16:34:32.968231 22262 raft_consensus.cc:729] T 57bce21fbf7d4a13aea368cd77d33e2f P 6a190cafd1f94f38ad1c6bf176f8066b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a190cafd1f94f38ad1c6bf176f8066b, State: Initialized, Role: FOLLOWER
I20260926 16:34:32.968889 22263 consensus_queue.cc:260] T 57bce21fbf7d4a13aea368cd77d33e2f P f1f4f18f75344a569ccd2bf27a10f8d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5eb6546b9642e5b64b22da3a5298ca" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 32913 } } peers { permanent_uuid: "6a190cafd1f94f38ad1c6bf176f8066b" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45075 } } peers { permanent_uuid: "f1f4f18f75344a569ccd2bf27a10f8d7" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 41569 } }
I20260926 16:34:32.968943 22262 consensus_queue.cc:260] T 57bce21fbf7d4a13aea368cd77d33e2f P 6a190cafd1f94f38ad1c6bf176f8066b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5eb6546b9642e5b64b22da3a5298ca" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 32913 } } peers { permanent_uuid: "6a190cafd1f94f38ad1c6bf176f8066b" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45075 } } peers { permanent_uuid: "f1f4f18f75344a569ccd2bf27a10f8d7" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 41569 } }
I20260926 16:34:32.969216 22111 heartbeater.cc:503] Master 127.6.33.62:38429 was elected leader, sending a full tablet report...
I20260926 16:34:32.969969 22264 ts_tablet_manager.cc:1428] T 57bce21fbf7d4a13aea368cd77d33e2f P fa5eb6546b9642e5b64b22da3a5298ca: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260926 16:34:32.971710 22263 ts_tablet_manager.cc:1428] T 57bce21fbf7d4a13aea368cd77d33e2f P f1f4f18f75344a569ccd2bf27a10f8d7: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260926 16:34:32.971750 22262 ts_tablet_manager.cc:1428] T 57bce21fbf7d4a13aea368cd77d33e2f P 6a190cafd1f94f38ad1c6bf176f8066b: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260926 16:34:32.972318 22244 heartbeater.cc:503] Master 127.6.33.62:38429 was elected leader, sending a full tablet report...
W20260926 16:34:33.044428 21978 tablet.cc:2369] T 57bce21fbf7d4a13aea368cd77d33e2f P f1f4f18f75344a569ccd2bf27a10f8d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 16:34:33.060901 22112 tablet.cc:2369] T 57bce21fbf7d4a13aea368cd77d33e2f P fa5eb6546b9642e5b64b22da3a5298ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 16:34:33.074018 22245 tablet.cc:2369] T 57bce21fbf7d4a13aea368cd77d33e2f P 6a190cafd1f94f38ad1c6bf176f8066b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:34:33.096354 22268 raft_consensus.cc:482] T 57bce21fbf7d4a13aea368cd77d33e2f P fa5eb6546b9642e5b64b22da3a5298ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:34:33.096689 22268 raft_consensus.cc:504] T 57bce21fbf7d4a13aea368cd77d33e2f P fa5eb6546b9642e5b64b22da3a5298ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5eb6546b9642e5b64b22da3a5298ca" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 32913 } } peers { permanent_uuid: "6a190cafd1f94f38ad1c6bf176f8066b" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45075 } } peers { permanent_uuid: "f1f4f18f75344a569ccd2bf27a10f8d7" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 41569 } }
I20260926 16:34:33.098644 22268 leader_election.cc:288] T 57bce21fbf7d4a13aea368cd77d33e2f P fa5eb6546b9642e5b64b22da3a5298ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a190cafd1f94f38ad1c6bf176f8066b (127.6.33.3:45075), f1f4f18f75344a569ccd2bf27a10f8d7 (127.6.33.1:41569)
I20260926 16:34:33.106038 22261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:33.099176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:37055 (local address 127.6.33.3:45075)
0926 16:34:33.099392 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:33.099398 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:33.100144 (+ 746us) server_negotiation.cc:407] Connection header received
0926 16:34:33.100303 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:33.100308 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:33.100389 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:33.100491 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:34:33.101137 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:33.101875 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:33.104188 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:33.105049 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:33.105143 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:33.105237 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:33.105642 (+ 405us) server_negotiation.cc:299] Negotiation successful
0926 16:34:33.105769 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 16:34:33.106060 22259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:33.099157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:57387 (local address 127.6.33.1:41569)
0926 16:34:33.099392 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:33.099399 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:33.099834 (+ 435us) server_negotiation.cc:407] Connection header received
0926 16:34:33.100038 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:33.100044 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:33.100137 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:33.100235 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:34:33.100934 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:33.101726 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:33.104227 (+ 2501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:33.105144 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:33.105287 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:33.105393 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:33.105676 (+ 283us) server_negotiation.cc:299] Negotiation successful
0926 16:34:33.105806 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260926 16:34:33.106184 22271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:33.098890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:41569 (local address 127.6.33.2:57387)
0926 16:34:33.099697 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:33.099719 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 16:34:33.099891 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:34:33.100279 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:34:33.100284 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:34:33.100310 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:34:33.100743 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:33.100800 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:33.101865 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:33.101870 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:33.104052 (+ 2182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:33.104064 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:33.105375 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:33.105379 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:33.105504 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:33.105525 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 16:34:33.105632 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 16:34:33.105757 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"mutex_wait_us":8,"thread_start_us":425,"threads_started":1}
I20260926 16:34:33.106619 22272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:33.098950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.3:45075 (local address 127.6.33.2:37055)
0926 16:34:33.100011 (+ 1061us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:33.100034 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 16:34:33.100169 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:34:33.100514 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:34:33.100517 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:34:33.100536 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:34:33.100971 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:33.100984 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:33.102008 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:33.102015 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:33.104022 (+ 2007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:33.104035 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:33.105259 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:33.105265 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:33.105475 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:33.105505 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 16:34:33.105580 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 16:34:33.105725 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":876,"mutex_wait_us":54,"thread_start_us":627,"threads_started":1}
I20260926 16:34:33.107288 21932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57bce21fbf7d4a13aea368cd77d33e2f" candidate_uuid: "fa5eb6546b9642e5b64b22da3a5298ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1f4f18f75344a569ccd2bf27a10f8d7" is_pre_election: true
I20260926 16:34:33.107669 22199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57bce21fbf7d4a13aea368cd77d33e2f" candidate_uuid: "fa5eb6546b9642e5b64b22da3a5298ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a190cafd1f94f38ad1c6bf176f8066b" is_pre_election: true
I20260926 16:34:33.107807 21932 raft_consensus.cc:2454] T 57bce21fbf7d4a13aea368cd77d33e2f P f1f4f18f75344a569ccd2bf27a10f8d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa5eb6546b9642e5b64b22da3a5298ca in term 0.
I20260926 16:34:33.108132 22199 raft_consensus.cc:2454] T 57bce21fbf7d4a13aea368cd77d33e2f P 6a190cafd1f94f38ad1c6bf176f8066b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa5eb6546b9642e5b64b22da3a5298ca in term 0.
I20260926 16:34:33.108503 21999 leader_election.cc:302] T 57bce21fbf7d4a13aea368cd77d33e2f P fa5eb6546b9642e5b64b22da3a5298ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1f4f18f75344a569ccd2bf27a10f8d7, fa5eb6546b9642e5b64b22da3a5298ca; no voters:
I20260926 16:34:33.108961 22268 raft_consensus.cc:2781] T 57bce21fbf7d4a13aea368cd77d33e2f P fa5eb6546b9642e5b64b22da3a5298ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:34:33.109107 22268 raft_consensus.cc:482] T 57bce21fbf7d4a13aea368cd77d33e2f P fa5eb6546b9642e5b64b22da3a5298ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:34:33.109215 22268 raft_consensus.cc:3037] T 57bce21fbf7d4a13aea368cd77d33e2f P fa5eb6546b9642e5b64b22da3a5298ca [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:34:33.111442 22268 raft_consensus.cc:504] T 57bce21fbf7d4a13aea368cd77d33e2f P fa5eb6546b9642e5b64b22da3a5298ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5eb6546b9642e5b64b22da3a5298ca" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 32913 } } peers { permanent_uuid: "6a190cafd1f94f38ad1c6bf176f8066b" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45075 } } peers { permanent_uuid: "f1f4f18f75344a569ccd2bf27a10f8d7" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 41569 } }
I20260926 16:34:33.112226 22268 leader_election.cc:288] T 57bce21fbf7d4a13aea368cd77d33e2f P fa5eb6546b9642e5b64b22da3a5298ca [CANDIDATE]: Term 1 election: Requested vote from peers 6a190cafd1f94f38ad1c6bf176f8066b (127.6.33.3:45075), f1f4f18f75344a569ccd2bf27a10f8d7 (127.6.33.1:41569)
I20260926 16:34:33.112668 22199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57bce21fbf7d4a13aea368cd77d33e2f" candidate_uuid: "fa5eb6546b9642e5b64b22da3a5298ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a190cafd1f94f38ad1c6bf176f8066b"
I20260926 16:34:33.112829 21932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57bce21fbf7d4a13aea368cd77d33e2f" candidate_uuid: "fa5eb6546b9642e5b64b22da3a5298ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1f4f18f75344a569ccd2bf27a10f8d7"
I20260926 16:34:33.112963 22199 raft_consensus.cc:3037] T 57bce21fbf7d4a13aea368cd77d33e2f P 6a190cafd1f94f38ad1c6bf176f8066b [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:34:33.113061 21932 raft_consensus.cc:3037] T 57bce21fbf7d4a13aea368cd77d33e2f P f1f4f18f75344a569ccd2bf27a10f8d7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:34:33.115242 21932 raft_consensus.cc:2454] T 57bce21fbf7d4a13aea368cd77d33e2f P f1f4f18f75344a569ccd2bf27a10f8d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa5eb6546b9642e5b64b22da3a5298ca in term 1.
I20260926 16:34:33.115280 22199 raft_consensus.cc:2454] T 57bce21fbf7d4a13aea368cd77d33e2f P 6a190cafd1f94f38ad1c6bf176f8066b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa5eb6546b9642e5b64b22da3a5298ca in term 1.
I20260926 16:34:33.115626 21999 leader_election.cc:302] T 57bce21fbf7d4a13aea368cd77d33e2f P fa5eb6546b9642e5b64b22da3a5298ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1f4f18f75344a569ccd2bf27a10f8d7, fa5eb6546b9642e5b64b22da3a5298ca; no voters:
I20260926 16:34:33.116062 22268 raft_consensus.cc:2781] T 57bce21fbf7d4a13aea368cd77d33e2f P fa5eb6546b9642e5b64b22da3a5298ca [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:34:33.116456 22268 raft_consensus.cc:686] T 57bce21fbf7d4a13aea368cd77d33e2f P fa5eb6546b9642e5b64b22da3a5298ca [term 1 LEADER]: Becoming Leader. State: Replica: fa5eb6546b9642e5b64b22da3a5298ca, State: Running, Role: LEADER
I20260926 16:34:33.117053 22268 consensus_queue.cc:237] T 57bce21fbf7d4a13aea368cd77d33e2f P fa5eb6546b9642e5b64b22da3a5298ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5eb6546b9642e5b64b22da3a5298ca" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 32913 } } peers { permanent_uuid: "6a190cafd1f94f38ad1c6bf176f8066b" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45075 } } peers { permanent_uuid: "f1f4f18f75344a569ccd2bf27a10f8d7" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 41569 } }
I20260926 16:34:33.124836 21786 catalog_manager.cc:5495] T 57bce21fbf7d4a13aea368cd77d33e2f P fa5eb6546b9642e5b64b22da3a5298ca reported cstate change: term changed from 0 to 1, leader changed from <none> to fa5eb6546b9642e5b64b22da3a5298ca (127.6.33.2). New cstate: current_term: 1 leader_uuid: "fa5eb6546b9642e5b64b22da3a5298ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5eb6546b9642e5b64b22da3a5298ca" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 32913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a190cafd1f94f38ad1c6bf176f8066b" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1f4f18f75344a569ccd2bf27a10f8d7" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 41569 } health_report { overall_health: UNKNOWN } } }
I20260926 16:34:33.168610 22257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:33.160959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38026 (local address 127.6.33.2:32913)
0926 16:34:33.161178 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:33.161185 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:33.161205 (+ 20us) server_negotiation.cc:407] Connection header received
0926 16:34:33.161314 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:33.161328 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:33.161436 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:33.161624 (+ 188us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:34:33.162149 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:33.163289 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:33.165796 (+ 2507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:33.166455 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:33.166588 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:33.167309 (+ 721us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:34:33.167970 (+ 661us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:34:33.168109 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:33.168230 (+ 121us) server_negotiation.cc:299] Negotiation successful
0926 16:34:33.168328 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 16:34:33.177335 22199 raft_consensus.cc:1261] T 57bce21fbf7d4a13aea368cd77d33e2f P 6a190cafd1f94f38ad1c6bf176f8066b [term 1 FOLLOWER]: Refusing update from remote peer fa5eb6546b9642e5b64b22da3a5298ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:34:33.177335 21932 raft_consensus.cc:1261] T 57bce21fbf7d4a13aea368cd77d33e2f P f1f4f18f75344a569ccd2bf27a10f8d7 [term 1 FOLLOWER]: Refusing update from remote peer fa5eb6546b9642e5b64b22da3a5298ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:34:33.178350 22273 consensus_queue.cc:1035] T 57bce21fbf7d4a13aea368cd77d33e2f P fa5eb6546b9642e5b64b22da3a5298ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a190cafd1f94f38ad1c6bf176f8066b" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:34:33.178692 22268 consensus_queue.cc:1035] T 57bce21fbf7d4a13aea368cd77d33e2f P fa5eb6546b9642e5b64b22da3a5298ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1f4f18f75344a569ccd2bf27a10f8d7" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 41569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:34:33.191794 22278 mvcc.cc:205] Tried to move back new op lower bound from 7333644178124136448 to 7333644177891643392. Current Snapshot: MvccSnapshot[applied={T|T < 7333644178124136448}]
I20260926 16:34:33.192389 22277 mvcc.cc:205] Tried to move back new op lower bound from 7333644178124136448 to 7333644177891643392. Current Snapshot: MvccSnapshot[applied={T|T < 7333644178124136448}]
I20260926 16:34:33.206492 21786 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:34:33.206812 21786 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:34:33.210421 21786 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3: Sending DeleteTablet for 3 replicas of tablet 57bce21fbf7d4a13aea368cd77d33e2f
I20260926 16:34:33.211711 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 21849
I20260926 16:34:33.211690 21912 tablet_service.cc:1514] Processing DeleteTablet for tablet 57bce21fbf7d4a13aea368cd77d33e2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:34:33 UTC) from {username='kudu', principal='kudu/127.6.33.62@KRBTEST.COM'} at 127.0.0.1:56720
I20260926 16:34:33.211884 22046 tablet_service.cc:1514] Processing DeleteTablet for tablet 57bce21fbf7d4a13aea368cd77d33e2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:34:33 UTC) from {username='kudu', principal='kudu/127.6.33.62@KRBTEST.COM'} at 127.0.0.1:38010
I20260926 16:34:33.211933 22179 tablet_service.cc:1514] Processing DeleteTablet for tablet 57bce21fbf7d4a13aea368cd77d33e2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:34:33 UTC) from {username='kudu', principal='kudu/127.6.33.62@KRBTEST.COM'} at 127.0.0.1:37464
I20260926 16:34:33.225178 22286 tablet_replica.cc:330] T 57bce21fbf7d4a13aea368cd77d33e2f P f1f4f18f75344a569ccd2bf27a10f8d7: stopping tablet replica
I20260926 16:34:33.225785 22287 tablet_replica.cc:330] T 57bce21fbf7d4a13aea368cd77d33e2f P fa5eb6546b9642e5b64b22da3a5298ca: stopping tablet replica
I20260926 16:34:33.226400 22287 raft_consensus.cc:2229] T 57bce21fbf7d4a13aea368cd77d33e2f P fa5eb6546b9642e5b64b22da3a5298ca [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:34:33.226967 22286 raft_consensus.cc:2229] T 57bce21fbf7d4a13aea368cd77d33e2f P f1f4f18f75344a569ccd2bf27a10f8d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:34:33.227734 22287 raft_consensus.cc:2258] T 57bce21fbf7d4a13aea368cd77d33e2f P fa5eb6546b9642e5b64b22da3a5298ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:34:33.228395 22286 raft_consensus.cc:2258] T 57bce21fbf7d4a13aea368cd77d33e2f P f1f4f18f75344a569ccd2bf27a10f8d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:34:33.231223 22286 ts_tablet_manager.cc:1905] T 57bce21fbf7d4a13aea368cd77d33e2f P f1f4f18f75344a569ccd2bf27a10f8d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:34:33.232622 22288 tablet_replica.cc:330] T 57bce21fbf7d4a13aea368cd77d33e2f P 6a190cafd1f94f38ad1c6bf176f8066b: stopping tablet replica
I20260926 16:34:33.233100 22288 raft_consensus.cc:2229] T 57bce21fbf7d4a13aea368cd77d33e2f P 6a190cafd1f94f38ad1c6bf176f8066b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:34:33.233471 22287 ts_tablet_manager.cc:1905] T 57bce21fbf7d4a13aea368cd77d33e2f P fa5eb6546b9642e5b64b22da3a5298ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:34:33.233635 22288 raft_consensus.cc:2258] T 57bce21fbf7d4a13aea368cd77d33e2f P 6a190cafd1f94f38ad1c6bf176f8066b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:34:33.237334 22288 ts_tablet_manager.cc:1905] T 57bce21fbf7d4a13aea368cd77d33e2f P 6a190cafd1f94f38ad1c6bf176f8066b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:34:33.239431 22286 ts_tablet_manager.cc:1918] T 57bce21fbf7d4a13aea368cd77d33e2f P f1f4f18f75344a569ccd2bf27a10f8d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:34:33.239602 22286 log.cc:1196] T 57bce21fbf7d4a13aea368cd77d33e2f P f1f4f18f75344a569ccd2bf27a10f8d7: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-0/wal/wals/57bce21fbf7d4a13aea368cd77d33e2f
I20260926 16:34:33.240125 22286 ts_tablet_manager.cc:1939] T 57bce21fbf7d4a13aea368cd77d33e2f P f1f4f18f75344a569ccd2bf27a10f8d7: Deleting consensus metadata
I20260926 16:34:33.242046 22287 ts_tablet_manager.cc:1918] T 57bce21fbf7d4a13aea368cd77d33e2f P fa5eb6546b9642e5b64b22da3a5298ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:34:33.242153 21772 catalog_manager.cc:4841] TS f1f4f18f75344a569ccd2bf27a10f8d7 (127.6.33.1:41569): tablet 57bce21fbf7d4a13aea368cd77d33e2f (table test-table [id=ea06c49cf1b54c9398d1d755a84e58da]) successfully deleted
I20260926 16:34:33.242391 22287 log.cc:1196] T 57bce21fbf7d4a13aea368cd77d33e2f P fa5eb6546b9642e5b64b22da3a5298ca: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-1/wal/wals/57bce21fbf7d4a13aea368cd77d33e2f
I20260926 16:34:33.243026 22287 ts_tablet_manager.cc:1939] T 57bce21fbf7d4a13aea368cd77d33e2f P fa5eb6546b9642e5b64b22da3a5298ca: Deleting consensus metadata
I20260926 16:34:33.244539 22288 ts_tablet_manager.cc:1918] T 57bce21fbf7d4a13aea368cd77d33e2f P 6a190cafd1f94f38ad1c6bf176f8066b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:34:33.244696 22288 log.cc:1196] T 57bce21fbf7d4a13aea368cd77d33e2f P 6a190cafd1f94f38ad1c6bf176f8066b: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-2/wal/wals/57bce21fbf7d4a13aea368cd77d33e2f
I20260926 16:34:33.244952 21775 catalog_manager.cc:4841] TS fa5eb6546b9642e5b64b22da3a5298ca (127.6.33.2:32913): tablet 57bce21fbf7d4a13aea368cd77d33e2f (table test-table [id=ea06c49cf1b54c9398d1d755a84e58da]) successfully deleted
I20260926 16:34:33.245221 22288 ts_tablet_manager.cc:1939] T 57bce21fbf7d4a13aea368cd77d33e2f P 6a190cafd1f94f38ad1c6bf176f8066b: Deleting consensus metadata
Sep 26 16:34:33 dist-test-slave-khd3 krb5kdc[21743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.1@KRBTEST.COM
I20260926 16:34:33.247110 21775 catalog_manager.cc:4841] TS 6a190cafd1f94f38ad1c6bf176f8066b (127.6.33.3:45075): tablet 57bce21fbf7d4a13aea368cd77d33e2f (table test-table [id=ea06c49cf1b54c9398d1d755a84e58da]) successfully deleted
I20260926 16:34:33.254534 22259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:33.212891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56734 (local address 127.6.33.1:41569)
0926 16:34:33.213122 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:33.213131 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:33.223921 (+ 10790us) server_negotiation.cc:407] Connection header received
0926 16:34:33.224514 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:33.224523 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:33.224651 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:33.224776 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:34:33.227446 (+ 2670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:33.229116 (+ 1670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:33.234033 (+ 4917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:33.234703 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:33.234815 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:33.247352 (+ 12537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:34:33.247369 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:34:33.247388 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:34:33.247437 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:34:33.251419 (+ 3982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:33.252195 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:33.252205 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:33.252209 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:33.252382 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:33.252741 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:33.252746 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:33.252750 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:33.253253 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:34:33.253548 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:33.253844 (+ 296us) server_negotiation.cc:299] Negotiation successful
0926 16:34:33.254082 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 16:34:33.255267 21972 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:34:33.346652 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 21849
I20260926 16:34:33.364430 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 21983
Sep 26 16:34:33 dist-test-slave-khd3 krb5kdc[21743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.2@KRBTEST.COM
I20260926 16:34:33.383159 22257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:33.365215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38030 (local address 127.6.33.2:32913)
0926 16:34:33.365397 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:33.365401 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:33.365479 (+ 78us) server_negotiation.cc:407] Connection header received
0926 16:34:33.365666 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:33.365670 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:33.365735 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:33.365818 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:34:33.367521 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:33.368332 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:33.370447 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:33.371031 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:33.371140 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:33.376666 (+ 5526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:34:33.376678 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:34:33.376693 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:34:33.376732 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:34:33.380053 (+ 3321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:33.381055 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:33.381061 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:33.381064 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:33.381202 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:33.381608 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:33.381611 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:33.381613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:33.381981 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:34:33.382230 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:33.382622 (+ 392us) server_negotiation.cc:299] Negotiation successful
0926 16:34:33.382786 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 16:34:33.383899 22106 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:34:33.498715 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 21983
I20260926 16:34:33.517445 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 22116
Sep 26 16:34:33 dist-test-slave-khd3 krb5kdc[21743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.3@KRBTEST.COM
I20260926 16:34:33.533813 22261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:33.518282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37478 (local address 127.6.33.3:45075)
0926 16:34:33.518494 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:33.518502 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:33.518547 (+ 45us) server_negotiation.cc:407] Connection header received
0926 16:34:33.518838 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:33.518842 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:33.518910 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:33.519034 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:34:33.520317 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:33.521105 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:33.522514 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:33.522918 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:33.523066 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:33.527962 (+ 4896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:34:33.527979 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:34:33.527995 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:34:33.528033 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:34:33.531028 (+ 2995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:33.531787 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:33.531796 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:33.531799 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:33.531944 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:33.532278 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:33.532282 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:33.532284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:33.532633 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:34:33.532920 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:33.533274 (+ 354us) server_negotiation.cc:299] Negotiation successful
0926 16:34:33.533470 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 16:34:33.534466 22239 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:34:33.627843 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 22116
I20260926 16:34:33.644819 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 21756
I20260926 16:34:33.646067 21818 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:34:33.760474 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 21756
I20260926 16:34:53.782246 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:38429
--webserver_interface=127.6.33.62
--webserver_port=36315
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:39917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.33.62:38429
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:34:54.120030 22294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:34:54.120347 22294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:34:54.120441 22294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:34:54.127899 22294 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 16:34:54.128021 22294 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 16:34:54.128060 22294 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 16:34:54.128105 22294 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:34:54.128155 22294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:34:54.128225 22294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:34:54.128263 22294 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:34:54.128314 22294 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:34:54.138729 22294 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:39917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.6.33.62:38429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:38429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.33.62
--webserver_port=36315
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:34:54.140030 22294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:34:54.141328 22294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:34:54.165557 22303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:34:54.165513 22301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:34:54.165580 22300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:34:54.167151 22294 server_base.cc:1034] running on GCE node
I20260926 16:34:54.168246 22294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:34:54.170061 22294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:34:54.171386 22294 hybrid_clock.cc:648] HybridClock initialized: now 1790440494171306 us; error 52 us; skew 500 ppm
I20260926 16:34:54.171830 22294 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:34:54 dist-test-slave-khd3 krb5kdc[21743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440494, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:34:54.175906 22294 init.cc:377] Logged in from keytab as kudu/127.6.33.62@KRBTEST.COM (short username kudu)
I20260926 16:34:54.178210 22294 webserver.cc:460] Webserver started at http://127.6.33.62:36315/ using document root <none> and password file <none>
I20260926 16:34:54.179147 22294 fs_manager.cc:359] Metadata directory not provided
I20260926 16:34:54.179318 22294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:34:54.186120 22294 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 16:34:54.189498 22310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:34:54.191147 22294 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:34:54.191378 22294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/master-0/wal
uuid: "5856adf7a90443d69e7c54ca18a824a3"
format_stamp: "Formatted at 2026-09-26 16:34:31 on dist-test-slave-khd3"
server_key: "592738d743631c0b10b072f0afdfd34d"
server_key_iv: "a103f64f7d5298cbe1fee8d4a4d0b3a9"
server_key_version: "encryptionkey@0"
I20260926 16:34:54.192549 22294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:34:54.219471 22294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:34:54.224435 22294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:34:54.224846 22294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:34:54.245157 22294 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.62:38429
I20260926 16:34:54.245200 22361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.62:38429 every 8 connection(s)
I20260926 16:34:54.246804 22294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
I20260926 16:34:54.254053 22362 sys_catalog.cc:263] Verifying existing consensus state
I20260926 16:34:54.257352 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 22294
I20260926 16:34:54.257886 22362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3: Bootstrap starting.
I20260926 16:34:54.258419 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.1:41569
--local_ip_for_outbound_sockets=127.6.33.1
--tserver_master_addrs=127.6.33.62:38429
--webserver_port=41389
--webserver_interface=127.6.33.1
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab
--principal=kudu/127.6.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:39917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:34:54.281108 22362 log.cc:824] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3: Log is configured to *not* fsync() on all Append() calls
I20260926 16:34:54.293661 22362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 16:34:54.294265 22362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3: Bootstrap complete.
I20260926 16:34:54.303810 22362 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5856adf7a90443d69e7c54ca18a824a3" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 38429 } }
I20260926 16:34:54.304666 22362 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5856adf7a90443d69e7c54ca18a824a3, State: Initialized, Role: FOLLOWER
I20260926 16:34:54.305405 22362 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5856adf7a90443d69e7c54ca18a824a3" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 38429 } }
I20260926 16:34:54.305711 22362 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:34:54.305840 22362 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:34:54.306017 22362 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3 [term 1 FOLLOWER]: Advancing to term 2
I20260926 16:34:54.310057 22362 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5856adf7a90443d69e7c54ca18a824a3" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 38429 } }
I20260926 16:34:54.310632 22362 leader_election.cc:302] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5856adf7a90443d69e7c54ca18a824a3; no voters:
I20260926 16:34:54.311388 22362 leader_election.cc:288] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 16:34:54.311523 22366 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 16:34:54.312819 22366 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3 [term 2 LEADER]: Becoming Leader. State: Replica: 5856adf7a90443d69e7c54ca18a824a3, State: Running, Role: LEADER
I20260926 16:34:54.313494 22366 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5856adf7a90443d69e7c54ca18a824a3" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 38429 } }
I20260926 16:34:54.313936 22362 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:34:54.316143 22368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5856adf7a90443d69e7c54ca18a824a3. Latest consensus state: current_term: 2 leader_uuid: "5856adf7a90443d69e7c54ca18a824a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5856adf7a90443d69e7c54ca18a824a3" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 38429 } } }
I20260926 16:34:54.316546 22368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3 [sys.catalog]: This master's current role is: LEADER
I20260926 16:34:54.317029 22367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5856adf7a90443d69e7c54ca18a824a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5856adf7a90443d69e7c54ca18a824a3" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 38429 } } }
I20260926 16:34:54.317370 22367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3 [sys.catalog]: This master's current role is: LEADER
I20260926 16:34:54.326471 22372 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:34:54.336632 22372 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:34:54.338717 22372 catalog_manager.cc:1259] Loaded cluster ID: dcb4b0b790d34c829828b3ff2f70dc6e
I20260926 16:34:54.338924 22372 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:34:54.344421 22372 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:34:54.346689 22372 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3: Loaded TSK: 0
I20260926 16:34:54.360504 22383 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 16:34:54.378417 22372 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3: Generated new TSK 1
I20260926 16:34:54.378832 22372 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 16:34:54.652002 22364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:34:54.652403 22364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:34:54.652594 22364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:34:54.663118 22364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:34:54.663286 22364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:34:54.663514 22364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.1
I20260926 16:34:54.674396 22364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:39917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab
--principal=kudu/127.6.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.1:41569
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.33.1
--webserver_port=41389
--tserver_master_addrs=127.6.33.62:38429
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.1
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:34:54.675907 22364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:34:54.677569 22364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:34:54.692935 22390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:34:54.693312 22389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:34:54.698127 22392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:34:54.703675 22364 server_base.cc:1034] running on GCE node
I20260926 16:34:54.704614 22364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:34:54.705893 22364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:34:54.707180 22364 hybrid_clock.cc:648] HybridClock initialized: now 1790440494707120 us; error 73 us; skew 500 ppm
I20260926 16:34:54.707564 22364 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:34:54 dist-test-slave-khd3 krb5kdc[21743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440494, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:34:54.710994 22364 init.cc:377] Logged in from keytab as kudu/127.6.33.1@KRBTEST.COM (short username kudu)
I20260926 16:34:54.712718 22364 webserver.cc:460] Webserver started at http://127.6.33.1:41389/ using document root <none> and password file <none>
I20260926 16:34:54.713567 22364 fs_manager.cc:359] Metadata directory not provided
I20260926 16:34:54.713806 22364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:34:54.719913 22364 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 16:34:54.723011 22399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:34:54.724570 22364 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 16:34:54.724808 22364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-0/wal
uuid: "f1f4f18f75344a569ccd2bf27a10f8d7"
format_stamp: "Formatted at 2026-09-26 16:34:31 on dist-test-slave-khd3"
server_key: "4389302381262e875ef68fd0d70de23a"
server_key_iv: "8f43022fb015dd8520e5a24fa782ed6f"
server_key_version: "encryptionkey@0"
I20260926 16:34:54.725939 22364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:34:54.746785 22364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:34:54.751282 22364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:34:54.751579 22364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:34:54.752710 22364 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:34:54.754218 22364 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:34:54.754343 22364 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:34:54.754487 22364 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:34:54.754575 22364 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:34:54.791610 22364 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.1:41569
I20260926 16:34:54.791702 22511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.1:41569 every 8 connection(s)
I20260926 16:34:54.793182 22364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
I20260926 16:34:54.797387 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 22364
I20260926 16:34:54.798205 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.2:32913
--local_ip_for_outbound_sockets=127.6.33.2
--tserver_master_addrs=127.6.33.62:38429
--webserver_port=38901
--webserver_interface=127.6.33.2
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab
--principal=kudu/127.6.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:39917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 16:34:54 dist-test-slave-khd3 krb5kdc[21743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440494, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.1@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:34:54.821365 22514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:54.796088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:39217 (local address 127.6.33.62:38429)
0926 16:34:54.796720 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:54.796741 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:54.800574 (+ 3833us) server_negotiation.cc:407] Connection header received
0926 16:34:54.801026 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:54.801047 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:54.801523 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:54.801936 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:34:54.803476 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:54.805629 (+ 2153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:54.808451 (+ 2822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:54.809192 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:54.809405 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:54.813354 (+ 3949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:34:54.813362 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:34:54.813378 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:34:54.813423 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:34:54.816517 (+ 3094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:54.817392 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:54.817423 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:54.817429 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:54.817593 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:54.818069 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:54.818079 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:54.818082 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:54.818566 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:34:54.818781 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:54.819597 (+ 816us) server_negotiation.cc:299] Negotiation successful
0926 16:34:54.820015 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":237,"threads_started":1}
I20260926 16:34:54.821365 22515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:54.796406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:38429 (local address 127.6.33.1:39217)
0926 16:34:54.797360 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:54.797445 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0926 16:34:54.800188 (+ 2743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:34:54.802090 (+ 1902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:34:54.802098 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:34:54.802639 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:34:54.803203 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:54.803222 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:54.805865 (+ 2643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:54.805873 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:54.808191 (+ 2318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:54.808214 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:54.809466 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:54.809474 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:54.809746 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:54.810482 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:34:54.810509 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:34:54.813023 (+ 2514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:34:54.816701 (+ 3678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:34:54.816713 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:34:54.816735 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:34:54.817242 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:34:54.817733 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:34:54.817738 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:34:54.817742 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:34:54.817942 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:34:54.818727 (+ 785us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:34:54.818736 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:34:54.819088 (+ 352us) client_negotiation.cc:769] Sending connection context
0926 16:34:54.819372 (+ 284us) client_negotiation.cc:240] Negotiation successful
0926 16:34:54.819959 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":401,"threads_started":1}
I20260926 16:34:54.824649 22512 heartbeater.cc:347] Connected to a master server at 127.6.33.62:38429
I20260926 16:34:54.825062 22512 heartbeater.cc:464] Registering TS with master...
I20260926 16:34:54.825999 22512 heartbeater.cc:511] Master 127.6.33.62:38429 requested a full tablet report, sending...
I20260926 16:34:54.828543 22327 ts_manager.cc:194] Registered new tserver with Master: f1f4f18f75344a569ccd2bf27a10f8d7 (127.6.33.1:41569)
I20260926 16:34:54.830202 22327 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.33.1@KRBTEST.COM'} at 127.6.33.1:39217
W20260926 16:34:55.142191 22516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:34:55.142462 22516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:34:55.142553 22516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:34:55.150095 22516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:34:55.150204 22516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:34:55.150367 22516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.2
I20260926 16:34:55.160355 22516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:39917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab
--principal=kudu/127.6.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.2:32913
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.33.2
--webserver_port=38901
--tserver_master_addrs=127.6.33.62:38429
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.2
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:34:55.161885 22516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:34:55.163305 22516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:34:55.177240 22522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:34:55.177374 22521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:34:55.177357 22524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:34:55.178397 22516 server_base.cc:1034] running on GCE node
I20260926 16:34:55.179064 22516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:34:55.180114 22516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:34:55.181315 22516 hybrid_clock.cc:648] HybridClock initialized: now 1790440495181263 us; error 42 us; skew 500 ppm
I20260926 16:34:55.181649 22516 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:34:55 dist-test-slave-khd3 krb5kdc[21743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440495, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:34:55.185070 22516 init.cc:377] Logged in from keytab as kudu/127.6.33.2@KRBTEST.COM (short username kudu)
I20260926 16:34:55.186571 22516 webserver.cc:460] Webserver started at http://127.6.33.2:38901/ using document root <none> and password file <none>
I20260926 16:34:55.187260 22516 fs_manager.cc:359] Metadata directory not provided
I20260926 16:34:55.187413 22516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:34:55.192826 22516 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260926 16:34:55.195767 22531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:34:55.197414 22516 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 16:34:55.197602 22516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-1/wal
uuid: "fa5eb6546b9642e5b64b22da3a5298ca"
format_stamp: "Formatted at 2026-09-26 16:34:32 on dist-test-slave-khd3"
server_key: "45135926b847a97d0c7f4a5c30a68dd2"
server_key_iv: "f68ea020c7bf848a946f845f2a8e7e12"
server_key_version: "encryptionkey@0"
I20260926 16:34:55.198698 22516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:34:55.231038 22516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:34:55.235685 22516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:34:55.235984 22516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:34:55.237044 22516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:34:55.238427 22516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:34:55.238559 22516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:34:55.238698 22516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:34:55.238791 22516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:34:55.274616 22516 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.2:32913
I20260926 16:34:55.274712 22643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.2:32913 every 8 connection(s)
I20260926 16:34:55.276171 22516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
I20260926 16:34:55.276635 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 22516
I20260926 16:34:55.277539 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.3:45075
--local_ip_for_outbound_sockets=127.6.33.3
--tserver_master_addrs=127.6.33.62:38429
--webserver_port=45539
--webserver_interface=127.6.33.3
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab
--principal=kudu/127.6.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:39917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 16:34:55 dist-test-slave-khd3 krb5kdc[21743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440495, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.2@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:34:55.300943 22514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:55.279156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:42257 (local address 127.6.33.62:38429)
0926 16:34:55.279353 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:55.279360 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:55.280836 (+ 1476us) server_negotiation.cc:407] Connection header received
0926 16:34:55.283561 (+ 2725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:55.283569 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:55.283731 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:55.283853 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:34:55.285887 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:55.287113 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:55.288750 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:55.289543 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:55.289693 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:55.293312 (+ 3619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:34:55.293321 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:34:55.293329 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:34:55.293379 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:34:55.297306 (+ 3927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:55.298164 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:55.298172 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:55.298179 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:55.298349 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:55.298798 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:55.298805 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:55.298811 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:55.299268 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:34:55.299446 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:55.300145 (+ 699us) server_negotiation.cc:299] Negotiation successful
0926 16:34:55.300444 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 16:34:55.301714 22646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:55.279763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:38429 (local address 127.6.33.2:42257)
0926 16:34:55.280637 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:55.280682 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 16:34:55.283202 (+ 2520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:34:55.284109 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:34:55.284122 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:34:55.284892 (+ 770us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:34:55.285681 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:55.285702 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:55.287291 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:55.287295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:55.288563 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:55.288575 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:55.289763 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:55.289767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:55.289974 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:55.290703 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:34:55.290726 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:34:55.293039 (+ 2313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:34:55.297514 (+ 4475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:34:55.297522 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:34:55.297537 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:34:55.298004 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:34:55.298468 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:34:55.298471 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:34:55.298473 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:34:55.298668 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:34:55.299399 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:34:55.299406 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:34:55.299746 (+ 340us) client_negotiation.cc:769] Sending connection context
0926 16:34:55.299969 (+ 223us) client_negotiation.cc:240] Negotiation successful
0926 16:34:55.300390 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":331,"threads_started":1}
I20260926 16:34:55.303449 22644 heartbeater.cc:347] Connected to a master server at 127.6.33.62:38429
I20260926 16:34:55.303791 22644 heartbeater.cc:464] Registering TS with master...
I20260926 16:34:55.304538 22644 heartbeater.cc:511] Master 127.6.33.62:38429 requested a full tablet report, sending...
I20260926 16:34:55.306080 22327 ts_manager.cc:194] Registered new tserver with Master: fa5eb6546b9642e5b64b22da3a5298ca (127.6.33.2:32913)
I20260926 16:34:55.307200 22327 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.33.2@KRBTEST.COM'} at 127.6.33.2:42257
W20260926 16:34:55.613994 22647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:34:55.614296 22647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:34:55.614393 22647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:34:55.622184 22647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:34:55.622359 22647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:34:55.622594 22647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.3
I20260926 16:34:55.635514 22647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:39917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab
--principal=kudu/127.6.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.3:45075
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.33.3
--webserver_port=45539
--tserver_master_addrs=127.6.33.62:38429
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.3
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:34:55.636850 22647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:34:55.638190 22647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:34:55.651913 22652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:34:55.651983 22653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:34:55.652493 22655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:34:55.652839 22647 server_base.cc:1034] running on GCE node
I20260926 16:34:55.653461 22647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:34:55.654503 22647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:34:55.655714 22647 hybrid_clock.cc:648] HybridClock initialized: now 1790440495655672 us; error 36 us; skew 500 ppm
I20260926 16:34:55.656051 22647 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:34:55 dist-test-slave-khd3 krb5kdc[21743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440495, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:34:55.659330 22647 init.cc:377] Logged in from keytab as kudu/127.6.33.3@KRBTEST.COM (short username kudu)
I20260926 16:34:55.660951 22647 webserver.cc:460] Webserver started at http://127.6.33.3:45539/ using document root <none> and password file <none>
I20260926 16:34:55.661660 22647 fs_manager.cc:359] Metadata directory not provided
I20260926 16:34:55.661779 22647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:34:55.667694 22647 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 16:34:55.670575 22662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:34:55.672073 22647 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 16:34:55.672281 22647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-2/wal
uuid: "6a190cafd1f94f38ad1c6bf176f8066b"
format_stamp: "Formatted at 2026-09-26 16:34:32 on dist-test-slave-khd3"
server_key: "0beebb995bbc9497045f52e93f40e1f8"
server_key_iv: "d69056b5ed03938b92bee3a940adc843"
server_key_version: "encryptionkey@0"
I20260926 16:34:55.673502 22647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:34:55.700639 22647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:34:55.705672 22647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:34:55.705964 22647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:34:55.706977 22647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:34:55.708293 22647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:34:55.708387 22647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:34:55.708523 22647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:34:55.708642 22647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:34:55.743278 22647 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.3:45075
I20260926 16:34:55.743330 22774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.3:45075 every 8 connection(s)
I20260926 16:34:55.744738 22647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
I20260926 16:34:55.754842 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 22647
Sep 26 16:34:55 dist-test-slave-khd3 krb5kdc[21743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440495, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.3@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:34:55.768272 22514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:55.747481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.3:55707 (local address 127.6.33.62:38429)
0926 16:34:55.747691 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:55.747696 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:55.748922 (+ 1226us) server_negotiation.cc:407] Connection header received
0926 16:34:55.750991 (+ 2069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:55.750996 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:55.751070 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:55.751163 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:34:55.752856 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:55.753571 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:55.755312 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:55.755857 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:55.756000 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:55.760530 (+ 4530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:34:55.760537 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:34:55.760542 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:34:55.760588 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:34:55.764539 (+ 3951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:55.765408 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:55.765414 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:55.765419 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:55.765599 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:55.766039 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:55.766046 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:55.766052 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:55.766476 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:34:55.766639 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:55.767643 (+ 1004us) server_negotiation.cc:299] Negotiation successful
0926 16:34:55.767872 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 16:34:55.769484 22777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:55.747947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:38429 (local address 127.6.33.3:55707)
0926 16:34:55.748670 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:55.748720 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 16:34:55.750688 (+ 1968us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:34:55.751424 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:34:55.751433 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:34:55.752019 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:34:55.752627 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:55.752647 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:55.753744 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:55.753748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:55.755158 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:55.755170 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:55.756107 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:55.756114 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:55.756446 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:55.757261 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:34:55.757291 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:34:55.760239 (+ 2948us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:34:55.764716 (+ 4477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:34:55.764725 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:34:55.764741 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:34:55.765276 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:34:55.765718 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:34:55.765721 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:34:55.765723 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:34:55.765910 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:34:55.766654 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:34:55.766663 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:34:55.767166 (+ 503us) client_negotiation.cc:769] Sending connection context
0926 16:34:55.767534 (+ 368us) client_negotiation.cc:240] Negotiation successful
0926 16:34:55.768140 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":294,"threads_started":1}
I20260926 16:34:55.771355 22775 heartbeater.cc:347] Connected to a master server at 127.6.33.62:38429
I20260926 16:34:55.771827 22775 heartbeater.cc:464] Registering TS with master...
I20260926 16:34:55.772887 22775 heartbeater.cc:511] Master 127.6.33.62:38429 requested a full tablet report, sending...
I20260926 16:34:55.773974 22779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:55.755929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36060 (local address 127.6.33.62:38429)
0926 16:34:55.756557 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:55.756563 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:55.756988 (+ 425us) server_negotiation.cc:407] Connection header received
0926 16:34:55.757180 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:55.757187 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:55.757268 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:55.757411 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:34:55.759298 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:55.760424 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:55.762637 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:55.763178 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:55.763282 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:55.766747 (+ 3465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:34:55.766754 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:34:55.766758 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:34:55.766793 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:34:55.770315 (+ 3522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:55.771345 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:55.771353 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:55.771357 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:55.771519 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:55.772248 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:55.772254 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:55.772257 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:55.772653 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:34:55.773019 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:55.773311 (+ 292us) server_negotiation.cc:299] Negotiation successful
0926 16:34:55.773574 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":357,"threads_started":1}
I20260926 16:34:55.774812 22327 ts_manager.cc:194] Registered new tserver with Master: 6a190cafd1f94f38ad1c6bf176f8066b (127.6.33.3:45075)
I20260926 16:34:55.775903 22327 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.33.3@KRBTEST.COM'} at 127.6.33.3:55707
I20260926 16:34:55.776847 6276 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:34:55.784417 22779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:55.777942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36074 (local address 127.6.33.62:38429)
0926 16:34:55.778672 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:55.778678 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:55.778698 (+ 20us) server_negotiation.cc:407] Connection header received
0926 16:34:55.778962 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:55.778966 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:55.779121 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:55.779288 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:34:55.779766 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:55.780559 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:55.782219 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:55.782713 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:55.782872 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:55.783418 (+ 546us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:34:55.783646 (+ 228us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0926 16:34:55.784085 (+ 439us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36074: token expired
Metrics: {"server-negotiator.queue_time_us":562}
W20260926 16:34:55.784592 22779 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36074: token expired
W20260926 16:34:55.785073 22247 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.6.33.62:38429)
I20260926 16:34:55.799839 22779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:55.785775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36090 (local address 127.6.33.62:38429)
0926 16:34:55.785945 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:55.785950 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:55.785965 (+ 15us) server_negotiation.cc:407] Connection header received
0926 16:34:55.786006 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:55.786009 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:55.786065 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:55.786154 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:34:55.787558 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:55.788295 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:55.789941 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:55.790359 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:55.790455 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:55.793461 (+ 3006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:34:55.793467 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:34:55.793471 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:34:55.793515 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:34:55.797081 (+ 3566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:55.798008 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:55.798012 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:55.798014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:55.798158 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:55.798514 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:55.798517 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:55.798519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:55.798833 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:34:55.799010 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:55.799311 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 16:34:55.799489 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 16:34:55.804265 22326 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36090:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:34:55.814330 22326 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:34:55.834003 22512 heartbeater.cc:503] Master 127.6.33.62:38429 was elected leader, sending a full tablet report...
I20260926 16:34:55.843453 22785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:55.827488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.2:32913 (local address 127.0.0.1:54842)
0926 16:34:55.828475 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:55.828520 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 16:34:55.828747 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:34:55.835087 (+ 6340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:34:55.835094 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:34:55.835123 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:34:55.835580 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:55.835595 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:55.837906 (+ 2311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:55.837911 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:55.840462 (+ 2551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:55.840477 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:55.842312 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:55.842318 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:55.842556 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:55.842634 (+ 78us) client_negotiation.cc:769] Sending connection context
0926 16:34:55.842768 (+ 134us) client_negotiation.cc:240] Negotiation successful
0926 16:34:55.842987 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"mutex_wait_us":3,"thread_start_us":467,"threads_started":1}
I20260926 16:34:55.843509 22790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:55.829278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59806 (local address 127.6.33.3:45075)
0926 16:34:55.830371 (+ 1093us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:55.830384 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:55.834836 (+ 4452us) server_negotiation.cc:407] Connection header received
0926 16:34:55.835150 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:55.835159 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:55.835418 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:55.835591 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:34:55.836655 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:55.838291 (+ 1636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:55.841061 (+ 2770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:55.842053 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:55.842182 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:55.842448 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:55.842947 (+ 499us) server_negotiation.cc:299] Negotiation successful
0926 16:34:55.843136 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":850,"thread_start_us":615,"threads_started":1}
I20260926 16:34:55.843791 22787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:55.828351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50484 (local address 127.6.33.1:41569)
0926 16:34:55.833423 (+ 5072us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:55.833436 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:55.834312 (+ 876us) server_negotiation.cc:407] Connection header received
0926 16:34:55.835095 (+ 783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:55.835103 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:55.835362 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:55.835548 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:34:55.836537 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:55.837751 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:55.840395 (+ 2644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:55.842060 (+ 1665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:55.842209 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:55.842748 (+ 539us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:55.843282 (+ 534us) server_negotiation.cc:299] Negotiation successful
0926 16:34:55.843464 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4881,"thread_start_us":4747,"threads_started":1}
I20260926 16:34:55.844338 22788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:55.828542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54842 (local address 127.6.33.2:32913)
0926 16:34:55.834306 (+ 5764us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:55.834314 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:55.834340 (+ 26us) server_negotiation.cc:407] Connection header received
0926 16:34:55.834471 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:55.834480 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:55.834799 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:55.835793 (+ 994us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:34:55.836022 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:55.837745 (+ 1723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:55.840648 (+ 2903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:55.842069 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:55.843575 (+ 1506us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:55.843793 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:55.843858 (+ 65us) server_negotiation.cc:299] Negotiation successful
0926 16:34:55.844042 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5355,"thread_start_us":5257,"threads_started":1}
I20260926 16:34:55.844676 22789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:55.828838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.3:45075 (local address 127.0.0.1:59806)
0926 16:34:55.834711 (+ 5873us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:55.834730 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 16:34:55.834963 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:34:55.835945 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:34:55.835949 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:34:55.835972 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:34:55.836434 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:55.836448 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:55.838773 (+ 2325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:55.838778 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:55.840895 (+ 2117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:55.840905 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:55.842537 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:55.842542 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:55.842721 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:55.842752 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 16:34:55.844162 (+ 1410us) client_negotiation.cc:240] Negotiation successful
0926 16:34:55.844316 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5761,"thread_start_us":5561,"threads_started":1}
I20260926 16:34:55.844915 22786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:55.828080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:41569 (local address 127.0.0.1:50484)
0926 16:34:55.834178 (+ 6098us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:55.834204 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 16:34:55.834884 (+ 680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:34:55.835866 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:34:55.835872 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:34:55.835898 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:34:55.836302 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:55.836311 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:55.837896 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:55.837905 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:55.840210 (+ 2305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:55.840231 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:55.842885 (+ 2654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:55.842891 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:55.843063 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:55.843095 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 16:34:55.843208 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 16:34:55.843336 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5968,"mutex_wait_us":43,"spinlock_wait_cycles":3840,"thread_start_us":5469,"threads_started":1}
I20260926 16:34:55.848459 22710 tablet_service.cc:1467] Processing CreateTablet for tablet 84928aa4779f4f1ea6e32a26c4b3d8b1 (DEFAULT_TABLE table=test-table [id=ca1341f68f014beaa0d8ed92340f38ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:34:55.848440 22447 tablet_service.cc:1467] Processing CreateTablet for tablet 84928aa4779f4f1ea6e32a26c4b3d8b1 (DEFAULT_TABLE table=test-table [id=ca1341f68f014beaa0d8ed92340f38ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:34:55.848440 22579 tablet_service.cc:1467] Processing CreateTablet for tablet 84928aa4779f4f1ea6e32a26c4b3d8b1 (DEFAULT_TABLE table=test-table [id=ca1341f68f014beaa0d8ed92340f38ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:34:55.849819 22710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84928aa4779f4f1ea6e32a26c4b3d8b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:34:55.849885 22579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84928aa4779f4f1ea6e32a26c4b3d8b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:34:55.849893 22447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84928aa4779f4f1ea6e32a26c4b3d8b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:34:55.863175 22791 tablet_bootstrap.cc:492] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P 6a190cafd1f94f38ad1c6bf176f8066b: Bootstrap starting.
I20260926 16:34:55.865886 22792 tablet_bootstrap.cc:492] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P fa5eb6546b9642e5b64b22da3a5298ca: Bootstrap starting.
I20260926 16:34:55.868852 22791 tablet_bootstrap.cc:654] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P 6a190cafd1f94f38ad1c6bf176f8066b: Neither blocks nor log segments found. Creating new log.
I20260926 16:34:55.869552 22793 tablet_bootstrap.cc:492] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P f1f4f18f75344a569ccd2bf27a10f8d7: Bootstrap starting.
I20260926 16:34:55.869664 22792 tablet_bootstrap.cc:654] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P fa5eb6546b9642e5b64b22da3a5298ca: Neither blocks nor log segments found. Creating new log.
I20260926 16:34:55.870523 22791 log.cc:824] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P 6a190cafd1f94f38ad1c6bf176f8066b: Log is configured to *not* fsync() on all Append() calls
I20260926 16:34:55.870926 22792 log.cc:824] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P fa5eb6546b9642e5b64b22da3a5298ca: Log is configured to *not* fsync() on all Append() calls
I20260926 16:34:55.873363 22793 tablet_bootstrap.cc:654] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P f1f4f18f75344a569ccd2bf27a10f8d7: Neither blocks nor log segments found. Creating new log.
I20260926 16:34:55.873822 22791 tablet_bootstrap.cc:492] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P 6a190cafd1f94f38ad1c6bf176f8066b: No bootstrap required, opened a new log
I20260926 16:34:55.874119 22791 ts_tablet_manager.cc:1397] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P 6a190cafd1f94f38ad1c6bf176f8066b: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.001s
I20260926 16:34:55.874212 22792 tablet_bootstrap.cc:492] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P fa5eb6546b9642e5b64b22da3a5298ca: No bootstrap required, opened a new log
I20260926 16:34:55.874493 22792 ts_tablet_manager.cc:1397] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P fa5eb6546b9642e5b64b22da3a5298ca: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260926 16:34:55.874697 22793 log.cc:824] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P f1f4f18f75344a569ccd2bf27a10f8d7: Log is configured to *not* fsync() on all Append() calls
I20260926 16:34:55.878764 22793 tablet_bootstrap.cc:492] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P f1f4f18f75344a569ccd2bf27a10f8d7: No bootstrap required, opened a new log
I20260926 16:34:55.879084 22793 ts_tablet_manager.cc:1397] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P f1f4f18f75344a569ccd2bf27a10f8d7: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260926 16:34:55.882849 22791 raft_consensus.cc:348] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P 6a190cafd1f94f38ad1c6bf176f8066b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5eb6546b9642e5b64b22da3a5298ca" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 32913 } } peers { permanent_uuid: "6a190cafd1f94f38ad1c6bf176f8066b" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45075 } } peers { permanent_uuid: "f1f4f18f75344a569ccd2bf27a10f8d7" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 41569 } }
I20260926 16:34:55.883276 22791 raft_consensus.cc:374] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P 6a190cafd1f94f38ad1c6bf176f8066b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:34:55.883373 22791 raft_consensus.cc:729] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P 6a190cafd1f94f38ad1c6bf176f8066b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a190cafd1f94f38ad1c6bf176f8066b, State: Initialized, Role: FOLLOWER
I20260926 16:34:55.884073 22791 consensus_queue.cc:260] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P 6a190cafd1f94f38ad1c6bf176f8066b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5eb6546b9642e5b64b22da3a5298ca" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 32913 } } peers { permanent_uuid: "6a190cafd1f94f38ad1c6bf176f8066b" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45075 } } peers { permanent_uuid: "f1f4f18f75344a569ccd2bf27a10f8d7" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 41569 } }
I20260926 16:34:55.885243 22775 heartbeater.cc:503] Master 127.6.33.62:38429 was elected leader, sending a full tablet report...
I20260926 16:34:55.885836 22792 raft_consensus.cc:348] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P fa5eb6546b9642e5b64b22da3a5298ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5eb6546b9642e5b64b22da3a5298ca" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 32913 } } peers { permanent_uuid: "6a190cafd1f94f38ad1c6bf176f8066b" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45075 } } peers { permanent_uuid: "f1f4f18f75344a569ccd2bf27a10f8d7" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 41569 } }
I20260926 16:34:55.886310 22792 raft_consensus.cc:374] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P fa5eb6546b9642e5b64b22da3a5298ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:34:55.886399 22792 raft_consensus.cc:729] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P fa5eb6546b9642e5b64b22da3a5298ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa5eb6546b9642e5b64b22da3a5298ca, State: Initialized, Role: FOLLOWER
I20260926 16:34:55.887004 22792 consensus_queue.cc:260] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P fa5eb6546b9642e5b64b22da3a5298ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5eb6546b9642e5b64b22da3a5298ca" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 32913 } } peers { permanent_uuid: "6a190cafd1f94f38ad1c6bf176f8066b" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45075 } } peers { permanent_uuid: "f1f4f18f75344a569ccd2bf27a10f8d7" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 41569 } }
I20260926 16:34:55.888259 22791 ts_tablet_manager.cc:1428] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P 6a190cafd1f94f38ad1c6bf176f8066b: Time spent starting tablet: real 0.014s user 0.010s sys 0.003s
I20260926 16:34:55.889330 22644 heartbeater.cc:503] Master 127.6.33.62:38429 was elected leader, sending a full tablet report...
I20260926 16:34:55.890007 22792 ts_tablet_manager.cc:1428] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P fa5eb6546b9642e5b64b22da3a5298ca: Time spent starting tablet: real 0.015s user 0.014s sys 0.001s
I20260926 16:34:55.893039 22793 raft_consensus.cc:348] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P f1f4f18f75344a569ccd2bf27a10f8d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5eb6546b9642e5b64b22da3a5298ca" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 32913 } } peers { permanent_uuid: "6a190cafd1f94f38ad1c6bf176f8066b" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45075 } } peers { permanent_uuid: "f1f4f18f75344a569ccd2bf27a10f8d7" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 41569 } }
I20260926 16:34:55.893489 22793 raft_consensus.cc:374] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P f1f4f18f75344a569ccd2bf27a10f8d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:34:55.893602 22793 raft_consensus.cc:729] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P f1f4f18f75344a569ccd2bf27a10f8d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1f4f18f75344a569ccd2bf27a10f8d7, State: Initialized, Role: FOLLOWER
I20260926 16:34:55.894357 22793 consensus_queue.cc:260] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P f1f4f18f75344a569ccd2bf27a10f8d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5eb6546b9642e5b64b22da3a5298ca" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 32913 } } peers { permanent_uuid: "6a190cafd1f94f38ad1c6bf176f8066b" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45075 } } peers { permanent_uuid: "f1f4f18f75344a569ccd2bf27a10f8d7" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 41569 } }
I20260926 16:34:55.896220 22793 ts_tablet_manager.cc:1428] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P f1f4f18f75344a569ccd2bf27a10f8d7: Time spent starting tablet: real 0.017s user 0.016s sys 0.000s
I20260926 16:34:55.979490 22797 raft_consensus.cc:482] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P 6a190cafd1f94f38ad1c6bf176f8066b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:34:55.979803 22797 raft_consensus.cc:504] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P 6a190cafd1f94f38ad1c6bf176f8066b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5eb6546b9642e5b64b22da3a5298ca" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 32913 } } peers { permanent_uuid: "6a190cafd1f94f38ad1c6bf176f8066b" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45075 } } peers { permanent_uuid: "f1f4f18f75344a569ccd2bf27a10f8d7" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 41569 } }
I20260926 16:34:55.981753 22797 leader_election.cc:288] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P 6a190cafd1f94f38ad1c6bf176f8066b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa5eb6546b9642e5b64b22da3a5298ca (127.6.33.2:32913), f1f4f18f75344a569ccd2bf27a10f8d7 (127.6.33.1:41569)
I20260926 16:34:55.988065 22777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:55.982073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.2:32913 (local address 127.6.33.3:33991)
0926 16:34:55.982261 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:55.982283 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 16:34:55.982401 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:34:55.982748 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:34:55.982753 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:34:55.982786 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:34:55.982979 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:55.982988 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:55.984239 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:55.984242 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:55.986172 (+ 1930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:55.986188 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:55.987445 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:55.987449 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:55.987660 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:55.987686 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 16:34:55.987762 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 16:34:55.987855 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 16:34:55.988231 22788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:55.982196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.3:33991 (local address 127.6.33.2:32913)
0926 16:34:55.982402 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:55.982408 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:55.982433 (+ 25us) server_negotiation.cc:407] Connection header received
0926 16:34:55.982516 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:55.982519 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:55.982578 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:55.982718 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:34:55.983148 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:55.984101 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:55.986329 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:55.987287 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:55.987425 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:55.987522 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:55.987823 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 16:34:55.987961 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 16:34:55.989174 22800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:55.982092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:41569 (local address 127.6.33.3:56497)
0926 16:34:55.983172 (+ 1080us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:55.983187 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 16:34:55.983287 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:34:55.983596 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:34:55.983599 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:34:55.983618 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:34:55.983902 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:55.983909 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:55.985013 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:55.985018 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:55.986773 (+ 1755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:34:55.986783 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:34:55.988440 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:34:55.988446 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:34:55.988616 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:55.988653 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 16:34:55.988750 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 16:34:55.988901 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":979,"mutex_wait_us":378,"thread_start_us":375,"threads_started":1}
I20260926 16:34:55.989198 22787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:55.982209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.3:56497 (local address 127.6.33.1:41569)
0926 16:34:55.982402 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:55.982408 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:55.983290 (+ 882us) server_negotiation.cc:407] Connection header received
0926 16:34:55.983375 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:55.983380 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:55.983463 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:55.983558 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:34:55.984054 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:55.984877 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:55.986929 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:55.988057 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:55.988197 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:55.988312 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:55.988827 (+ 515us) server_negotiation.cc:299] Negotiation successful
0926 16:34:55.988968 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 16:34:55.989316 22599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84928aa4779f4f1ea6e32a26c4b3d8b1" candidate_uuid: "6a190cafd1f94f38ad1c6bf176f8066b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa5eb6546b9642e5b64b22da3a5298ca" is_pre_election: true
I20260926 16:34:55.989938 22599 raft_consensus.cc:2454] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P fa5eb6546b9642e5b64b22da3a5298ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a190cafd1f94f38ad1c6bf176f8066b in term 0.
I20260926 16:34:55.990360 22467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84928aa4779f4f1ea6e32a26c4b3d8b1" candidate_uuid: "6a190cafd1f94f38ad1c6bf176f8066b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1f4f18f75344a569ccd2bf27a10f8d7" is_pre_election: true
I20260926 16:34:55.990628 22666 leader_election.cc:302] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P 6a190cafd1f94f38ad1c6bf176f8066b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a190cafd1f94f38ad1c6bf176f8066b, fa5eb6546b9642e5b64b22da3a5298ca; no voters:
I20260926 16:34:55.990939 22467 raft_consensus.cc:2454] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P f1f4f18f75344a569ccd2bf27a10f8d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a190cafd1f94f38ad1c6bf176f8066b in term 0.
I20260926 16:34:55.991108 22797 raft_consensus.cc:2781] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P 6a190cafd1f94f38ad1c6bf176f8066b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:34:55.991282 22797 raft_consensus.cc:482] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P 6a190cafd1f94f38ad1c6bf176f8066b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:34:55.991379 22797 raft_consensus.cc:3037] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P 6a190cafd1f94f38ad1c6bf176f8066b [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:34:55.993866 22797 raft_consensus.cc:504] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P 6a190cafd1f94f38ad1c6bf176f8066b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5eb6546b9642e5b64b22da3a5298ca" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 32913 } } peers { permanent_uuid: "6a190cafd1f94f38ad1c6bf176f8066b" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45075 } } peers { permanent_uuid: "f1f4f18f75344a569ccd2bf27a10f8d7" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 41569 } }
I20260926 16:34:55.994529 22797 leader_election.cc:288] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P 6a190cafd1f94f38ad1c6bf176f8066b [CANDIDATE]: Term 1 election: Requested vote from peers fa5eb6546b9642e5b64b22da3a5298ca (127.6.33.2:32913), f1f4f18f75344a569ccd2bf27a10f8d7 (127.6.33.1:41569)
I20260926 16:34:55.994937 22599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84928aa4779f4f1ea6e32a26c4b3d8b1" candidate_uuid: "6a190cafd1f94f38ad1c6bf176f8066b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa5eb6546b9642e5b64b22da3a5298ca"
I20260926 16:34:55.995162 22599 raft_consensus.cc:3037] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P fa5eb6546b9642e5b64b22da3a5298ca [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:34:55.995294 22467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84928aa4779f4f1ea6e32a26c4b3d8b1" candidate_uuid: "6a190cafd1f94f38ad1c6bf176f8066b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1f4f18f75344a569ccd2bf27a10f8d7"
I20260926 16:34:55.995509 22467 raft_consensus.cc:3037] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P f1f4f18f75344a569ccd2bf27a10f8d7 [term 0 FOLLOWER]: Advancing to term 1
W20260926 16:34:55.996302 22776 tablet.cc:2369] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P 6a190cafd1f94f38ad1c6bf176f8066b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:34:55.998711 22467 raft_consensus.cc:2454] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P f1f4f18f75344a569ccd2bf27a10f8d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a190cafd1f94f38ad1c6bf176f8066b in term 1.
I20260926 16:34:55.998701 22599 raft_consensus.cc:2454] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P fa5eb6546b9642e5b64b22da3a5298ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a190cafd1f94f38ad1c6bf176f8066b in term 1.
I20260926 16:34:55.999239 22663 leader_election.cc:302] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P 6a190cafd1f94f38ad1c6bf176f8066b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a190cafd1f94f38ad1c6bf176f8066b, f1f4f18f75344a569ccd2bf27a10f8d7; no voters:
I20260926 16:34:55.999612 22797 raft_consensus.cc:2781] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P 6a190cafd1f94f38ad1c6bf176f8066b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:34:56.000113 22797 raft_consensus.cc:686] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P 6a190cafd1f94f38ad1c6bf176f8066b [term 1 LEADER]: Becoming Leader. State: Replica: 6a190cafd1f94f38ad1c6bf176f8066b, State: Running, Role: LEADER
I20260926 16:34:56.000651 22797 consensus_queue.cc:237] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P 6a190cafd1f94f38ad1c6bf176f8066b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5eb6546b9642e5b64b22da3a5298ca" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 32913 } } peers { permanent_uuid: "6a190cafd1f94f38ad1c6bf176f8066b" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45075 } } peers { permanent_uuid: "f1f4f18f75344a569ccd2bf27a10f8d7" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 41569 } }
I20260926 16:34:56.005764 22326 catalog_manager.cc:5495] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P 6a190cafd1f94f38ad1c6bf176f8066b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a190cafd1f94f38ad1c6bf176f8066b (127.6.33.3). New cstate: current_term: 1 leader_uuid: "6a190cafd1f94f38ad1c6bf176f8066b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5eb6546b9642e5b64b22da3a5298ca" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 32913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a190cafd1f94f38ad1c6bf176f8066b" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45075 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1f4f18f75344a569ccd2bf27a10f8d7" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 41569 } health_report { overall_health: UNKNOWN } } }
W20260926 16:34:56.028496 22645 tablet.cc:2369] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P fa5eb6546b9642e5b64b22da3a5298ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:34:56.042066 22790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:56.036748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59818 (local address 127.6.33.3:45075)
0926 16:34:56.036983 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:56.036988 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:56.037005 (+ 17us) server_negotiation.cc:407] Connection header received
0926 16:34:56.037091 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:56.037115 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:56.037198 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:56.037326 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:34:56.037680 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:56.038395 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:56.039783 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:56.040198 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:56.040297 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:56.040816 (+ 519us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:34:56.041377 (+ 561us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:34:56.041492 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:56.041720 (+ 228us) server_negotiation.cc:299] Negotiation successful
0926 16:34:56.041825 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
W20260926 16:34:56.045683 22513 tablet.cc:2369] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P f1f4f18f75344a569ccd2bf27a10f8d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:34:56.051970 22467 raft_consensus.cc:1261] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P f1f4f18f75344a569ccd2bf27a10f8d7 [term 1 FOLLOWER]: Refusing update from remote peer 6a190cafd1f94f38ad1c6bf176f8066b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:34:56.051970 22599 raft_consensus.cc:1261] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P fa5eb6546b9642e5b64b22da3a5298ca [term 1 FOLLOWER]: Refusing update from remote peer 6a190cafd1f94f38ad1c6bf176f8066b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:34:56.053252 22797 consensus_queue.cc:1035] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P 6a190cafd1f94f38ad1c6bf176f8066b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa5eb6546b9642e5b64b22da3a5298ca" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 32913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:34:56.053682 22801 consensus_queue.cc:1035] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P 6a190cafd1f94f38ad1c6bf176f8066b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1f4f18f75344a569ccd2bf27a10f8d7" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 41569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:34:56.070575 22804 mvcc.cc:205] Tried to move back new op lower bound from 7333644271817617408 to 7333644271623557120. Current Snapshot: MvccSnapshot[applied={T|T < 7333644271817617408 or (T in {7333644271817617408})}]
I20260926 16:34:56.083880 22327 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:34:56.084189 22327 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:34:56.088093 22327 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3: Sending DeleteTablet for 3 replicas of tablet 84928aa4779f4f1ea6e32a26c4b3d8b1
I20260926 16:34:56.089434 22447 tablet_service.cc:1514] Processing DeleteTablet for tablet 84928aa4779f4f1ea6e32a26c4b3d8b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:34:56 UTC) from {username='kudu', principal='kudu/127.6.33.62@KRBTEST.COM'} at 127.0.0.1:50484
I20260926 16:34:56.089434 22579 tablet_service.cc:1514] Processing DeleteTablet for tablet 84928aa4779f4f1ea6e32a26c4b3d8b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:34:56 UTC) from {username='kudu', principal='kudu/127.6.33.62@KRBTEST.COM'} at 127.0.0.1:54842
I20260926 16:34:56.089867 22710 tablet_service.cc:1514] Processing DeleteTablet for tablet 84928aa4779f4f1ea6e32a26c4b3d8b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:34:56 UTC) from {username='kudu', principal='kudu/127.6.33.62@KRBTEST.COM'} at 127.0.0.1:59806
I20260926 16:34:56.090379 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 22364
I20260926 16:34:56.117028 22787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:56.091279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50492 (local address 127.6.33.1:41569)
0926 16:34:56.091856 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:56.091863 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:56.091883 (+ 20us) server_negotiation.cc:407] Connection header received
0926 16:34:56.091945 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:56.091951 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:56.092068 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:56.092180 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:34:56.094384 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:56.095797 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:56.098372 (+ 2575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:56.099117 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:56.099258 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:56.106754 (+ 7496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:34:56.106779 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:34:56.106800 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:34:56.106854 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:34:56.111146 (+ 4292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:56.112185 (+ 1039us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:56.112195 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:56.112199 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:56.112387 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:56.113670 (+ 1283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:56.113675 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:56.113679 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:56.114257 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:34:56.114570 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:56.116279 (+ 1709us) server_negotiation.cc:299] Negotiation successful
0926 16:34:56.116510 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438}
I20260926 16:34:56.117919 22507 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:34:56.122474 22815 tablet_replica.cc:330] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P 6a190cafd1f94f38ad1c6bf176f8066b: stopping tablet replica
I20260926 16:34:56.123010 22815 raft_consensus.cc:2229] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P 6a190cafd1f94f38ad1c6bf176f8066b [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:34:56.123728 22815 raft_consensus.cc:2258] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P 6a190cafd1f94f38ad1c6bf176f8066b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:34:56.124665 22814 tablet_replica.cc:330] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P fa5eb6546b9642e5b64b22da3a5298ca: stopping tablet replica
I20260926 16:34:56.125831 22814 raft_consensus.cc:2229] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P fa5eb6546b9642e5b64b22da3a5298ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:34:56.126374 22814 raft_consensus.cc:2258] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P fa5eb6546b9642e5b64b22da3a5298ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:34:56.127076 22815 ts_tablet_manager.cc:1905] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P 6a190cafd1f94f38ad1c6bf176f8066b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:34:56.128659 22814 ts_tablet_manager.cc:1905] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P fa5eb6546b9642e5b64b22da3a5298ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:34:56.135705 22814 ts_tablet_manager.cc:1918] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P fa5eb6546b9642e5b64b22da3a5298ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:34:56.135885 22814 log.cc:1196] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P fa5eb6546b9642e5b64b22da3a5298ca: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-1/wal/wals/84928aa4779f4f1ea6e32a26c4b3d8b1
I20260926 16:34:56.136461 22814 ts_tablet_manager.cc:1939] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P fa5eb6546b9642e5b64b22da3a5298ca: Deleting consensus metadata
I20260926 16:34:56.136948 22815 ts_tablet_manager.cc:1918] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P 6a190cafd1f94f38ad1c6bf176f8066b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:34:56.137145 22815 log.cc:1196] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P 6a190cafd1f94f38ad1c6bf176f8066b: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-2/wal/wals/84928aa4779f4f1ea6e32a26c4b3d8b1
I20260926 16:34:56.137742 22815 ts_tablet_manager.cc:1939] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P 6a190cafd1f94f38ad1c6bf176f8066b: Deleting consensus metadata
I20260926 16:34:56.138615 22314 catalog_manager.cc:4841] TS fa5eb6546b9642e5b64b22da3a5298ca (127.6.33.2:32913): tablet 84928aa4779f4f1ea6e32a26c4b3d8b1 (table test-table [id=ca1341f68f014beaa0d8ed92340f38ca]) successfully deleted
I20260926 16:34:56.139547 22314 catalog_manager.cc:4841] TS 6a190cafd1f94f38ad1c6bf176f8066b (127.6.33.3:45075): tablet 84928aa4779f4f1ea6e32a26c4b3d8b1 (table test-table [id=ca1341f68f014beaa0d8ed92340f38ca]) successfully deleted
I20260926 16:34:56.261297 22813 tablet_replica.cc:330] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P f1f4f18f75344a569ccd2bf27a10f8d7: stopping tablet replica
I20260926 16:34:56.261874 22813 raft_consensus.cc:2229] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P f1f4f18f75344a569ccd2bf27a10f8d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:34:56.262440 22813 raft_consensus.cc:2258] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P f1f4f18f75344a569ccd2bf27a10f8d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:34:56.262475 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 22364
I20260926 16:34:56.282795 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 22516
W20260926 16:34:56.282709 22311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.33.1:41569 (error 108)
I20260926 16:34:56.284679 22786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:56.284151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:41569 (local address 127.0.0.1:50502)
0926 16:34:56.284386 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:56.284508 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.1:41569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
W20260926 16:34:56.284924 22311 catalog_manager.cc:4584] TS f1f4f18f75344a569ccd2bf27a10f8d7 (127.6.33.1:41569): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 84928aa4779f4f1ea6e32a26c4b3d8b1: Network error: Client connection negotiation failed: client connection to 127.6.33.1:41569: connect: Connection refused (error 111)
I20260926 16:34:56.307022 22788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:56.283537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54858 (local address 127.6.33.2:32913)
0926 16:34:56.283718 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:56.283727 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:56.284053 (+ 326us) server_negotiation.cc:407] Connection header received
0926 16:34:56.284128 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:56.284133 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:56.284219 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:56.284318 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:34:56.286395 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:56.287518 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:56.290134 (+ 2616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:56.290789 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:56.290914 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:56.299602 (+ 8688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:34:56.299622 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:34:56.299641 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:34:56.299690 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:34:56.303699 (+ 4009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:56.304721 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:56.304731 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:56.304735 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:56.304915 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:56.305341 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:56.305349 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:56.305354 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:56.305836 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:34:56.306121 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:56.306447 (+ 326us) server_negotiation.cc:299] Negotiation successful
0926 16:34:56.306665 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 16:34:56.307957 22639 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:34:56.338788 22786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:56.338214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:41569 (local address 127.0.0.1:50982)
0926 16:34:56.338471 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:56.338593 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.1:41569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260926 16:34:56.339952 22786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:56.339540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:41569 (local address 127.0.0.1:50986)
0926 16:34:56.339709 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:56.339816 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.1:41569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 16:34:56.400497 22786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:56.399921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:41569 (local address 127.0.0.1:50990)
0926 16:34:56.400189 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:56.400306 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.1:41569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260926 16:34:56.401185 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 22516
I20260926 16:34:56.401960 22786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:56.401444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:41569 (local address 127.0.0.1:50992)
0926 16:34:56.401704 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:56.401808 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.1:41569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260926 16:34:56.423983 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 22647
I20260926 16:34:56.439023 22790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:56.424853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41356 (local address 127.6.33.3:45075)
0926 16:34:56.425061 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 16:34:56.425068 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:34:56.425149 (+ 81us) server_negotiation.cc:407] Connection header received
0926 16:34:56.425367 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:34:56.425377 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:34:56.425444 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:34:56.425568 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:34:56.426859 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:56.427733 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:56.429197 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:34:56.429681 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:34:56.429829 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:34:56.432534 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:34:56.432547 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:34:56.432571 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:34:56.432611 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:34:56.435856 (+ 3245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:56.436648 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:56.436661 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:56.436664 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:56.436878 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:34:56.437306 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:34:56.437311 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:34:56.437315 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:34:56.437724 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:34:56.438045 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0926 16:34:56.438377 (+ 332us) server_negotiation.cc:299] Negotiation successful
0926 16:34:56.438614 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 16:34:56.439704 22770 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:34:56.482750 22786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:56.482171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:41569 (local address 127.0.0.1:51004)
0926 16:34:56.482409 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:56.482521 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.1:41569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260926 16:34:56.483858 22786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:34:56.483411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:41569 (local address 127.0.0.1:51016)
0926 16:34:56.483631 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 16:34:56.483730 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.1:41569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260926 16:34:56.564960 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 22647
I20260926 16:34:56.584734 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 22294
I20260926 16:34:56.585940 22357 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:34:56.702255 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 22294
I20260926 16:35:16.724507 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:38429
--webserver_interface=127.6.33.62
--webserver_port=36315
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:39917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.33.62:38429
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:35:17.073688 22821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:35:17.073977 22821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:35:17.074069 22821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:35:17.081331 22821 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 16:35:17.081429 22821 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 16:35:17.081466 22821 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 16:35:17.081523 22821 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:35:17.081576 22821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:35:17.081643 22821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:35:17.081694 22821 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:35:17.081748 22821 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:35:17.092212 22821 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:39917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.6.33.62:38429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:38429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.33.62
--webserver_port=36315
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:35:17.093567 22821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:35:17.094877 22821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 16:35:17.107790 22821 server_base.cc:1034] running on GCE node
W20260926 16:35:17.107797 22829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:35:17.107853 22827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:35:17.107790 22826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:35:17.109193 22821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:35:17.110796 22821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:35:17.112085 22821 hybrid_clock.cc:648] HybridClock initialized: now 1790440517112009 us; error 42 us; skew 500 ppm
I20260926 16:35:17.112509 22821 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:35:17 dist-test-slave-khd3 krb5kdc[21743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440517, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:35:17.116044 22821 init.cc:377] Logged in from keytab as kudu/127.6.33.62@KRBTEST.COM (short username kudu)
I20260926 16:35:17.117697 22821 webserver.cc:460] Webserver started at http://127.6.33.62:36315/ using document root <none> and password file <none>
I20260926 16:35:17.118485 22821 fs_manager.cc:359] Metadata directory not provided
I20260926 16:35:17.118636 22821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:35:17.124516 22821 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 16:35:17.127614 22836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:35:17.129076 22821 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:35:17.129246 22821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/master-0/wal
uuid: "5856adf7a90443d69e7c54ca18a824a3"
format_stamp: "Formatted at 2026-09-26 16:34:31 on dist-test-slave-khd3"
server_key: "592738d743631c0b10b072f0afdfd34d"
server_key_iv: "a103f64f7d5298cbe1fee8d4a4d0b3a9"
server_key_version: "encryptionkey@0"
I20260926 16:35:17.130210 22821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:35:17.151186 22821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:35:17.155398 22821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:35:17.155723 22821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:35:17.175616 22821 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.62:38429
I20260926 16:35:17.175709 22887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.62:38429 every 8 connection(s)
I20260926 16:35:17.177309 22821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
I20260926 16:35:17.179522 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 22821
I20260926 16:35:17.180466 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.1:41569
--local_ip_for_outbound_sockets=127.6.33.1
--tserver_master_addrs=127.6.33.62:38429
--webserver_port=41389
--webserver_interface=127.6.33.1
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab
--principal=kudu/127.6.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:39917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:35:17.184928 22888 sys_catalog.cc:263] Verifying existing consensus state
I20260926 16:35:17.188226 22888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3: Bootstrap starting.
I20260926 16:35:17.209412 22888 log.cc:824] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3: Log is configured to *not* fsync() on all Append() calls
I20260926 16:35:17.225551 22888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 16:35:17.226138 22888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3: Bootstrap complete.
I20260926 16:35:17.235738 22888 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5856adf7a90443d69e7c54ca18a824a3" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 38429 } }
I20260926 16:35:17.236815 22888 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5856adf7a90443d69e7c54ca18a824a3, State: Initialized, Role: FOLLOWER
I20260926 16:35:17.237468 22888 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5856adf7a90443d69e7c54ca18a824a3" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 38429 } }
I20260926 16:35:17.237740 22888 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:35:17.237879 22888 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:35:17.238065 22888 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3 [term 2 FOLLOWER]: Advancing to term 3
I20260926 16:35:17.242102 22888 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5856adf7a90443d69e7c54ca18a824a3" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 38429 } }
I20260926 16:35:17.242619 22888 leader_election.cc:302] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5856adf7a90443d69e7c54ca18a824a3; no voters:
I20260926 16:35:17.243350 22888 leader_election.cc:288] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 16:35:17.243487 22892 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 16:35:17.244598 22892 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3 [term 3 LEADER]: Becoming Leader. State: Replica: 5856adf7a90443d69e7c54ca18a824a3, State: Running, Role: LEADER
I20260926 16:35:17.245285 22892 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5856adf7a90443d69e7c54ca18a824a3" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 38429 } }
I20260926 16:35:17.245625 22888 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:35:17.247781 22894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5856adf7a90443d69e7c54ca18a824a3. Latest consensus state: current_term: 3 leader_uuid: "5856adf7a90443d69e7c54ca18a824a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5856adf7a90443d69e7c54ca18a824a3" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 38429 } } }
I20260926 16:35:17.248149 22894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3 [sys.catalog]: This master's current role is: LEADER
I20260926 16:35:17.248803 22893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "5856adf7a90443d69e7c54ca18a824a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5856adf7a90443d69e7c54ca18a824a3" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 38429 } } }
I20260926 16:35:17.249115 22893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3 [sys.catalog]: This master's current role is: LEADER
I20260926 16:35:17.258672 22899 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:35:17.269876 22899 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:35:17.271957 22899 catalog_manager.cc:1259] Loaded cluster ID: dcb4b0b790d34c829828b3ff2f70dc6e
I20260926 16:35:17.272083 22899 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:35:17.277463 22899 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:35:17.279788 22899 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3: Loaded TSK: 0
I20260926 16:35:17.279968 22899 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5856adf7a90443d69e7c54ca18a824a3: Loaded TSK: 1
I20260926 16:35:17.280042 22909 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
F20260926 16:35:17.281461 22899 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fa16d4f69ad google::LogMessage::Fail() at ??:0
@ 0x7fa16d4f988a google::LogMessage::SendToLog() at ??:0
@ 0x7fa16d4f64de google::LogMessage::Flush() at ??:0
@ 0x7fa16d4f7a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fa17125a7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fa17124973e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fa182f251c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fa182f24783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fa182fb02a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fa182fb00c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fa182ffa5e9 std::function<>::operator()() at ??:0
@ 0x7fa182f28cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fa182f26c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fa182fae610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fa182fae439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fa170142345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fa17014591c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fa170145799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fa170120eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa1733fa6db start_thread at ??:0
@ 0x7fa16c39061f clone at ??:0
W20260926 16:35:17.588406 22890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:35:17.588711 22890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:35:17.588884 22890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:35:17.596477 22890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:35:17.596662 22890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:35:17.596892 22890 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.1
I20260926 16:35:17.607220 22890 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:39917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab
--principal=kudu/127.6.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.1:41569
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.33.1
--webserver_port=41389
--tserver_master_addrs=127.6.33.62:38429
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.1
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:35:17.608667 22890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:35:17.610113 22890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:35:17.625121 22936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:35:17.625151 22935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:35:17.625795 22938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:35:17.626457 22890 server_base.cc:1034] running on GCE node
I20260926 16:35:17.627225 22890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:35:17.628599 22890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:35:17.629829 22890 hybrid_clock.cc:648] HybridClock initialized: now 1790440517629747 us; error 64 us; skew 500 ppm
I20260926 16:35:17.630127 22890 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:35:17 dist-test-slave-khd3 krb5kdc[21743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440517, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:35:17.633431 22890 init.cc:377] Logged in from keytab as kudu/127.6.33.1@KRBTEST.COM (short username kudu)
I20260926 16:35:17.634965 22890 webserver.cc:460] Webserver started at http://127.6.33.1:41389/ using document root <none> and password file <none>
I20260926 16:35:17.635681 22890 fs_manager.cc:359] Metadata directory not provided
I20260926 16:35:17.635826 22890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:35:17.641449 22890 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.002s
I20260926 16:35:17.644326 22945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:35:17.645802 22890 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 16:35:17.645972 22890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-0/wal
uuid: "f1f4f18f75344a569ccd2bf27a10f8d7"
format_stamp: "Formatted at 2026-09-26 16:34:31 on dist-test-slave-khd3"
server_key: "4389302381262e875ef68fd0d70de23a"
server_key_iv: "8f43022fb015dd8520e5a24fa782ed6f"
server_key_version: "encryptionkey@0"
I20260926 16:35:17.646960 22890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:35:17.674549 22890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:35:17.679286 22890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:35:17.679586 22890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:35:17.680673 22890 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:35:17.682716 22952 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260926 16:35:17.689030 22890 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 16:35:17.689219 22890 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260926 16:35:17.689396 22890 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 16:35:17.692788 22890 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 16:35:17.692909 22890 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.000s sys 0.003s
I20260926 16:35:17.693382 22952 tablet_bootstrap.cc:492] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P f1f4f18f75344a569ccd2bf27a10f8d7: Bootstrap starting.
I20260926 16:35:17.723410 22952 log.cc:824] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P f1f4f18f75344a569ccd2bf27a10f8d7: Log is configured to *not* fsync() on all Append() calls
I20260926 16:35:17.733242 22890 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.1:41569
I20260926 16:35:17.733333 23059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.1:41569 every 8 connection(s)
I20260926 16:35:17.734923 22890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
I20260926 16:35:17.736814 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 22890
I20260926 16:35:17.737455 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.2:32913
--local_ip_for_outbound_sockets=127.6.33.2
--tserver_master_addrs=127.6.33.62:38429
--webserver_port=38901
--webserver_interface=127.6.33.2
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab
--principal=kudu/127.6.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:39917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:35:17.740124 23062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:17.738085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:38429 (local address 127.6.33.1:59475)
0926 16:35:17.738942 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:17.739188 (+ 246us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:38429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":371,"threads_started":1}
W20260926 16:35:17.742038 23060 heartbeater.cc:644] Failed to heartbeat to 127.6.33.62:38429 (0 consecutive failures): Network error: Failed to ping master at 127.6.33.62:38429: Client connection negotiation failed: client connection to 127.6.33.62:38429: connect: Connection refused (error 111)
I20260926 16:35:17.745210 23062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:17.742679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:38429 (local address 127.6.33.1:35047)
0926 16:35:17.744960 (+ 2281us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:17.745057 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:38429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2161}
I20260926 16:35:17.746251 23062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:17.745868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:38429 (local address 127.6.33.1:54621)
0926 16:35:17.746012 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:17.746098 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:38429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 16:35:17.746323 22952 tablet_bootstrap.cc:492] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P f1f4f18f75344a569ccd2bf27a10f8d7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
W20260926 16:35:17.746536 23060 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 16:35:17.746978 22952 tablet_bootstrap.cc:492] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P f1f4f18f75344a569ccd2bf27a10f8d7: Bootstrap complete.
I20260926 16:35:17.748572 22952 ts_tablet_manager.cc:1397] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P f1f4f18f75344a569ccd2bf27a10f8d7: Time spent bootstrapping tablet: real 0.056s user 0.034s sys 0.012s
I20260926 16:35:17.760102 22952 raft_consensus.cc:348] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P f1f4f18f75344a569ccd2bf27a10f8d7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5eb6546b9642e5b64b22da3a5298ca" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 32913 } } peers { permanent_uuid: "6a190cafd1f94f38ad1c6bf176f8066b" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45075 } } peers { permanent_uuid: "f1f4f18f75344a569ccd2bf27a10f8d7" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 41569 } }
I20260926 16:35:17.761854 22952 raft_consensus.cc:729] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P f1f4f18f75344a569ccd2bf27a10f8d7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1f4f18f75344a569ccd2bf27a10f8d7, State: Initialized, Role: FOLLOWER
I20260926 16:35:17.762825 22952 consensus_queue.cc:260] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P f1f4f18f75344a569ccd2bf27a10f8d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5eb6546b9642e5b64b22da3a5298ca" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 32913 } } peers { permanent_uuid: "6a190cafd1f94f38ad1c6bf176f8066b" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45075 } } peers { permanent_uuid: "f1f4f18f75344a569ccd2bf27a10f8d7" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 41569 } }
I20260926 16:35:17.764855 23062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:17.764401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:38429 (local address 127.6.33.1:43565)
0926 16:35:17.764578 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:17.764666 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:38429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 16:35:17.765187 22952 ts_tablet_manager.cc:1428] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P f1f4f18f75344a569ccd2bf27a10f8d7: Time spent starting tablet: real 0.016s user 0.018s sys 0.000s
I20260926 16:35:17.766213 23062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:17.765881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:38429 (local address 127.6.33.1:34945)
0926 16:35:17.766025 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:17.766098 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:38429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 16:35:18.085805 23063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:35:18.086097 23063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:35:18.086263 23063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:35:18.093616 23063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:35:18.093739 23063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:35:18.093936 23063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.2
I20260926 16:35:18.104017 23063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:39917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab
--principal=kudu/127.6.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.2:32913
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.33.2
--webserver_port=38901
--tserver_master_addrs=127.6.33.62:38429
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.2
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:35:18.105473 23063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:35:18.106854 23063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:35:18.120585 23069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:35:18.120639 23070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:35:18.120642 23072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:35:18.121635 23063 server_base.cc:1034] running on GCE node
I20260926 16:35:18.122365 23063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:35:18.123358 23063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:35:18.124599 23063 hybrid_clock.cc:648] HybridClock initialized: now 1790440518124546 us; error 48 us; skew 500 ppm
I20260926 16:35:18.125029 23063 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:35:18 dist-test-slave-khd3 krb5kdc[21743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440518, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:35:18.128212 23063 init.cc:377] Logged in from keytab as kudu/127.6.33.2@KRBTEST.COM (short username kudu)
I20260926 16:35:18.129807 23063 webserver.cc:460] Webserver started at http://127.6.33.2:38901/ using document root <none> and password file <none>
I20260926 16:35:18.130549 23063 fs_manager.cc:359] Metadata directory not provided
I20260926 16:35:18.130707 23063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:35:18.136508 23063 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260926 16:35:18.139465 23079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:35:18.141100 23063 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:35:18.141319 23063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-1/wal
uuid: "fa5eb6546b9642e5b64b22da3a5298ca"
format_stamp: "Formatted at 2026-09-26 16:34:32 on dist-test-slave-khd3"
server_key: "45135926b847a97d0c7f4a5c30a68dd2"
server_key_iv: "f68ea020c7bf848a946f845f2a8e7e12"
server_key_version: "encryptionkey@0"
I20260926 16:35:18.142428 23063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:35:18.168725 23063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:35:18.173151 23063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:35:18.173432 23063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:35:18.174465 23063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:35:18.175825 23063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:35:18.175956 23063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:35:18.176079 23063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:35:18.176152 23063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:35:18.210654 23063 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.2:32913
I20260926 16:35:18.210757 23191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.2:32913 every 8 connection(s)
I20260926 16:35:18.212131 23063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
I20260926 16:35:18.217118 23194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:18.215123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:38429 (local address 127.6.33.2:58147)
0926 16:35:18.215857 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:18.216069 (+ 212us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:38429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":281,"threads_started":1}
I20260926 16:35:18.217232 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 23063
W20260926 16:35:18.217921 23192 heartbeater.cc:644] Failed to heartbeat to 127.6.33.62:38429 (0 consecutive failures): Network error: Failed to ping master at 127.6.33.62:38429: Client connection negotiation failed: client connection to 127.6.33.62:38429: connect: Connection refused (error 111)
I20260926 16:35:18.218036 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.3:45075
--local_ip_for_outbound_sockets=127.6.33.3
--tserver_master_addrs=127.6.33.62:38429
--webserver_port=45539
--webserver_interface=127.6.33.3
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab
--principal=kudu/127.6.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:39917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:35:18.218824 23194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:18.218483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:38429 (local address 127.6.33.2:36985)
0926 16:35:18.218632 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:18.218711 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:38429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 16:35:18.219751 23194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:18.219441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:38429 (local address 127.6.33.2:58221)
0926 16:35:18.219574 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:18.219643 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:38429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 16:35:18.220124 23192 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 16:35:18.561884 23195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:35:18.562208 23195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:35:18.562419 23195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:35:18.570055 23195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:35:18.570227 23195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:35:18.570417 23195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.3
I20260926 16:35:18.580565 23195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:39917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab
--principal=kudu/127.6.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.3:45075
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.33.3
--webserver_port=45539
--tserver_master_addrs=127.6.33.62:38429
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.3
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:35:18.581933 23195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:35:18.583252 23195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:35:18.597151 23201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:35:18.597152 23203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:35:18.597388 23200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:35:18.597939 23195 server_base.cc:1034] running on GCE node
I20260926 16:35:18.598623 23195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:35:18.599664 23195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:35:18.600903 23195 hybrid_clock.cc:648] HybridClock initialized: now 1790440518600840 us; error 45 us; skew 500 ppm
I20260926 16:35:18.601291 23195 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:35:18 dist-test-slave-khd3 krb5kdc[21743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440518, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:35:18.609733 23195 init.cc:377] Logged in from keytab as kudu/127.6.33.3@KRBTEST.COM (short username kudu)
I20260926 16:35:18.611689 23195 webserver.cc:460] Webserver started at http://127.6.33.3:45539/ using document root <none> and password file <none>
I20260926 16:35:18.612500 23195 fs_manager.cc:359] Metadata directory not provided
I20260926 16:35:18.612679 23195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:35:18.618901 23195 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 16:35:18.622313 23210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:35:18.623857 23195 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 16:35:18.624058 23195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-2/wal
uuid: "6a190cafd1f94f38ad1c6bf176f8066b"
format_stamp: "Formatted at 2026-09-26 16:34:32 on dist-test-slave-khd3"
server_key: "0beebb995bbc9497045f52e93f40e1f8"
server_key_iv: "d69056b5ed03938b92bee3a940adc843"
server_key_version: "encryptionkey@0"
I20260926 16:35:18.625201 23195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:35:18.637596 23195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:35:18.642071 23195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:35:18.642361 23195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:35:18.643369 23195 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:35:18.644843 23195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:35:18.644980 23195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:35:18.645109 23195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:35:18.645208 23195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:35:18.681139 23195 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.3:45075
I20260926 16:35:18.681188 23322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.3:45075 every 8 connection(s)
I20260926 16:35:18.682710 23195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
I20260926 16:35:18.687744 23325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:18.685764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:38429 (local address 127.6.33.3:34439)
0926 16:35:18.686594 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:18.686812 (+ 218us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:38429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":347,"threads_started":1}
W20260926 16:35:18.688501 23323 heartbeater.cc:644] Failed to heartbeat to 127.6.33.62:38429 (0 consecutive failures): Network error: Failed to ping master at 127.6.33.62:38429: Client connection negotiation failed: client connection to 127.6.33.62:38429: connect: Connection refused (error 111)
I20260926 16:35:18.689540 23325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:18.689109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:38429 (local address 127.6.33.3:33193)
0926 16:35:18.689332 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:18.689417 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:38429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 16:35:18.690646 23325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:18.690247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:38429 (local address 127.6.33.3:37627)
0926 16:35:18.690419 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:18.690503 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:38429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260926 16:35:18.691020 23323 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 16:35:18.693149 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 23195
I20260926 16:35:18.768141 23327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:18.767236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:38429 (local address 127.6.33.1:41167)
0926 16:35:18.767756 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:18.767861 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:38429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":252,"threads_started":1}
I20260926 16:35:19.221745 23328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:19.220860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:38429 (local address 127.6.33.2:44149)
0926 16:35:19.221385 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:19.221482 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:38429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":234,"threads_started":1}
I20260926 16:35:19.478471 23329 raft_consensus.cc:482] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P f1f4f18f75344a569ccd2bf27a10f8d7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:35:19.478822 23329 raft_consensus.cc:504] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P f1f4f18f75344a569ccd2bf27a10f8d7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5eb6546b9642e5b64b22da3a5298ca" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 32913 } } peers { permanent_uuid: "6a190cafd1f94f38ad1c6bf176f8066b" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45075 } } peers { permanent_uuid: "f1f4f18f75344a569ccd2bf27a10f8d7" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 41569 } }
I20260926 16:35:19.480381 23329 leader_election.cc:288] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P f1f4f18f75344a569ccd2bf27a10f8d7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fa5eb6546b9642e5b64b22da3a5298ca (127.6.33.2:32913), 6a190cafd1f94f38ad1c6bf176f8066b (127.6.33.3:45075)
Sep 26 16:35:19 dist-test-slave-khd3 krb5kdc[21743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440517, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.1@KRBTEST.COM for kudu/127.6.33.2@KRBTEST.COM
Sep 26 16:35:19 dist-test-slave-khd3 krb5kdc[21743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440517, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.1@KRBTEST.COM for kudu/127.6.33.3@KRBTEST.COM
I20260926 16:35:19.505138 23333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:19.481970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:60193 (local address 127.6.33.3:45075)
0926 16:35:19.483551 (+ 1581us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:19.483565 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:19.483598 (+ 33us) server_negotiation.cc:407] Connection header received
0926 16:35:19.485982 (+ 2384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:19.485996 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:19.486536 (+ 540us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:19.487033 (+ 497us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:35:19.489171 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:19.490541 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:19.492443 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:19.493170 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:19.493349 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:19.497912 (+ 4563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:35:19.497923 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:35:19.497942 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:35:19.497992 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:35:19.501192 (+ 3200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:35:19.501954 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:35:19.501960 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:35:19.501966 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:35:19.502126 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:35:19.502604 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:35:19.502612 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:35:19.502615 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:35:19.503160 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:35:19.503339 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:19.504156 (+ 817us) server_negotiation.cc:299] Negotiation successful
0926 16:35:19.504664 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1412,"thread_start_us":1238,"threads_started":1}
I20260926 16:35:19.505141 23332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:19.481542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.3:45075 (local address 127.6.33.1:60193)
0926 16:35:19.482960 (+ 1418us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:19.482980 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 16:35:19.483092 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:35:19.487192 (+ 4100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:35:19.487203 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:35:19.488140 (+ 937us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:35:19.488970 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:19.488992 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:19.490743 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:19.490746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:19.492288 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:19.492301 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:19.493387 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:19.493394 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:19.493789 (+ 395us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:19.494605 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:35:19.494622 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:35:19.497563 (+ 2941us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:35:19.501328 (+ 3765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:35:19.501332 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:35:19.501334 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:35:19.501798 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:35:19.502251 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:35:19.502258 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:35:19.502266 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:35:19.502490 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:35:19.503319 (+ 829us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:35:19.503326 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:35:19.503651 (+ 325us) client_negotiation.cc:769] Sending connection context
0926 16:35:19.503934 (+ 283us) client_negotiation.cc:240] Negotiation successful
0926 16:35:19.504578 (+ 644us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1284,"mutex_wait_us":291,"spinlock_wait_cycles":12800,"thread_start_us":1115,"threads_started":1}
I20260926 16:35:19.505239 23330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:19.480824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.2:32913 (local address 127.6.33.1:43839)
0926 16:35:19.481422 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:19.481459 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 16:35:19.482120 (+ 661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:35:19.487191 (+ 5071us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:35:19.487202 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:35:19.488138 (+ 936us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:35:19.488970 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:19.488993 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:19.490677 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:19.490683 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:19.492293 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:19.492306 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:19.493432 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:19.493439 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:19.493707 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:19.494473 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:35:19.494516 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:35:19.497242 (+ 2726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:35:19.501175 (+ 3933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:35:19.501185 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:35:19.501202 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:35:19.501788 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:35:19.502256 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:35:19.502261 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:35:19.502264 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:35:19.502501 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:35:19.503369 (+ 868us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:35:19.503374 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:35:19.503650 (+ 276us) client_negotiation.cc:769] Sending connection context
0926 16:35:19.503930 (+ 280us) client_negotiation.cc:240] Negotiation successful
0926 16:35:19.504811 (+ 881us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"mutex_wait_us":555,"spinlock_wait_cycles":36096,"thread_start_us":276,"threads_started":1}
I20260926 16:35:19.505975 23331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:19.481112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:43839 (local address 127.6.33.2:32913)
0926 16:35:19.483319 (+ 2207us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:19.483338 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:19.483389 (+ 51us) server_negotiation.cc:407] Connection header received
0926 16:35:19.485984 (+ 2595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:19.485997 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:19.486536 (+ 539us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:19.487034 (+ 498us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:35:19.489176 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:19.490486 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:19.492453 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:19.493196 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:19.493438 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:19.497629 (+ 4191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:35:19.497641 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:35:19.497664 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:35:19.497721 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:35:19.501006 (+ 3285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:35:19.501949 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:35:19.501957 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:35:19.501965 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:35:19.502128 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:35:19.502650 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:35:19.502655 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:35:19.502658 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:35:19.503168 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:35:19.503357 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:19.504217 (+ 860us) server_negotiation.cc:299] Negotiation successful
0926 16:35:19.504798 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1992,"thread_start_us":1843,"threads_started":1}
W20260926 16:35:19.508683 22949 leader_election.cc:341] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P f1f4f18f75344a569ccd2bf27a10f8d7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6a190cafd1f94f38ad1c6bf176f8066b (127.6.33.3:45075): Not found: Tablet not found: 84928aa4779f4f1ea6e32a26c4b3d8b1
W20260926 16:35:19.509104 22949 leader_election.cc:341] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P f1f4f18f75344a569ccd2bf27a10f8d7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fa5eb6546b9642e5b64b22da3a5298ca (127.6.33.2:32913): Not found: Tablet not found: 84928aa4779f4f1ea6e32a26c4b3d8b1
I20260926 16:35:19.509253 22949 leader_election.cc:302] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P f1f4f18f75344a569ccd2bf27a10f8d7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1f4f18f75344a569ccd2bf27a10f8d7; no voters: 6a190cafd1f94f38ad1c6bf176f8066b, fa5eb6546b9642e5b64b22da3a5298ca
I20260926 16:35:19.509696 23329 raft_consensus.cc:2726] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P f1f4f18f75344a569ccd2bf27a10f8d7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 16:35:19.692466 23334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:19.691565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:38429 (local address 127.6.33.3:39017)
0926 16:35:19.692066 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:19.692209 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:38429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":250,"threads_started":1}
I20260926 16:35:19.769796 23332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:19.769381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:38429 (local address 127.6.33.1:50303)
0926 16:35:19.769548 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:19.769637 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:38429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260926 16:35:20.223385 23335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:20.222675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:38429 (local address 127.6.33.2:35061)
0926 16:35:20.223111 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:20.223190 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:38429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":248,"threads_started":1}
I20260926 16:35:20.694196 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:20.693447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:38429 (local address 127.6.33.3:39275)
0926 16:35:20.693905 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:20.693993 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:38429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":230,"threads_started":1}
I20260926 16:35:20.771590 23337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:20.770753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:38429 (local address 127.6.33.1:47003)
0926 16:35:20.771321 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:20.771408 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:38429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":237,"threads_started":1}
I20260926 16:35:21.225095 23338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:21.224321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:38429 (local address 127.6.33.2:44599)
0926 16:35:21.224841 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:21.224921 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:38429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":237,"threads_started":1}
I20260926 16:35:21.323689 23339 raft_consensus.cc:482] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P f1f4f18f75344a569ccd2bf27a10f8d7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:35:21.323884 23339 raft_consensus.cc:504] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P f1f4f18f75344a569ccd2bf27a10f8d7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5eb6546b9642e5b64b22da3a5298ca" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 32913 } } peers { permanent_uuid: "6a190cafd1f94f38ad1c6bf176f8066b" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45075 } } peers { permanent_uuid: "f1f4f18f75344a569ccd2bf27a10f8d7" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 41569 } }
I20260926 16:35:21.324599 23339 leader_election.cc:288] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P f1f4f18f75344a569ccd2bf27a10f8d7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fa5eb6546b9642e5b64b22da3a5298ca (127.6.33.2:32913), 6a190cafd1f94f38ad1c6bf176f8066b (127.6.33.3:45075)
W20260926 16:35:21.325769 22949 leader_election.cc:341] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P f1f4f18f75344a569ccd2bf27a10f8d7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fa5eb6546b9642e5b64b22da3a5298ca (127.6.33.2:32913): Not found: Tablet not found: 84928aa4779f4f1ea6e32a26c4b3d8b1
W20260926 16:35:21.325961 22949 leader_election.cc:341] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P f1f4f18f75344a569ccd2bf27a10f8d7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6a190cafd1f94f38ad1c6bf176f8066b (127.6.33.3:45075): Not found: Tablet not found: 84928aa4779f4f1ea6e32a26c4b3d8b1
I20260926 16:35:21.326035 22949 leader_election.cc:302] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P f1f4f18f75344a569ccd2bf27a10f8d7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1f4f18f75344a569ccd2bf27a10f8d7; no voters: 6a190cafd1f94f38ad1c6bf176f8066b, fa5eb6546b9642e5b64b22da3a5298ca
I20260926 16:35:21.326272 23339 raft_consensus.cc:2726] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P f1f4f18f75344a569ccd2bf27a10f8d7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 16:35:21.696175 23340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:21.695329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:38429 (local address 127.6.33.3:59733)
0926 16:35:21.695908 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:21.695995 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:38429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":301,"threads_started":1}
I20260926 16:35:21.773808 23341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:21.772826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:38429 (local address 127.6.33.1:55033)
0926 16:35:21.773371 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:21.773468 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:38429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":262,"threads_started":1}
I20260926 16:35:22.226824 23342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:22.226008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:38429 (local address 127.6.33.2:47815)
0926 16:35:22.226517 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:22.226614 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:38429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":280,"threads_started":1}
I20260926 16:35:22.697976 23343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:22.697212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:38429 (local address 127.6.33.3:33621)
0926 16:35:22.697694 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:22.697782 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:38429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":228,"threads_started":1}
I20260926 16:35:22.775764 23344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:22.775028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:38429 (local address 127.6.33.1:36035)
0926 16:35:22.775503 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:22.775592 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:38429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":252,"threads_started":1}
I20260926 16:35:23.202251 23345 raft_consensus.cc:482] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P f1f4f18f75344a569ccd2bf27a10f8d7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:35:23.202493 23345 raft_consensus.cc:504] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P f1f4f18f75344a569ccd2bf27a10f8d7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5eb6546b9642e5b64b22da3a5298ca" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 32913 } } peers { permanent_uuid: "6a190cafd1f94f38ad1c6bf176f8066b" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45075 } } peers { permanent_uuid: "f1f4f18f75344a569ccd2bf27a10f8d7" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 41569 } }
I20260926 16:35:23.203223 23345 leader_election.cc:288] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P f1f4f18f75344a569ccd2bf27a10f8d7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fa5eb6546b9642e5b64b22da3a5298ca (127.6.33.2:32913), 6a190cafd1f94f38ad1c6bf176f8066b (127.6.33.3:45075)
W20260926 16:35:23.204268 22949 leader_election.cc:341] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P f1f4f18f75344a569ccd2bf27a10f8d7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fa5eb6546b9642e5b64b22da3a5298ca (127.6.33.2:32913): Not found: Tablet not found: 84928aa4779f4f1ea6e32a26c4b3d8b1
W20260926 16:35:23.204439 22949 leader_election.cc:341] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P f1f4f18f75344a569ccd2bf27a10f8d7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6a190cafd1f94f38ad1c6bf176f8066b (127.6.33.3:45075): Not found: Tablet not found: 84928aa4779f4f1ea6e32a26c4b3d8b1
I20260926 16:35:23.204523 22949 leader_election.cc:302] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P f1f4f18f75344a569ccd2bf27a10f8d7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1f4f18f75344a569ccd2bf27a10f8d7; no voters: 6a190cafd1f94f38ad1c6bf176f8066b, fa5eb6546b9642e5b64b22da3a5298ca
I20260926 16:35:23.204794 23345 raft_consensus.cc:2726] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P f1f4f18f75344a569ccd2bf27a10f8d7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 16:35:23.228701 23346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:23.227917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:38429 (local address 127.6.33.2:57791)
0926 16:35:23.228423 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:23.228520 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:38429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":254,"threads_started":1}
I20260926 16:35:23.699690 23347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:23.698978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:38429 (local address 127.6.33.3:50895)
0926 16:35:23.699440 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:23.699526 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:38429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":251,"threads_started":1}
I20260926 16:35:23.777612 23348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:23.776838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:38429 (local address 127.6.33.1:33977)
0926 16:35:23.777317 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:23.777433 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:38429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":299,"threads_started":1}
I20260926 16:35:24.230623 23349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:24.229779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:38429 (local address 127.6.33.2:39433)
0926 16:35:24.230323 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:24.230411 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:38429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":309,"threads_started":1}
I20260926 16:35:24.701437 23350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:24.700658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:38429 (local address 127.6.33.3:59653)
0926 16:35:24.701167 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:24.701253 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:38429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":279,"threads_started":1}
I20260926 16:35:24.779621 23351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:24.778723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:38429 (local address 127.6.33.1:53367)
0926 16:35:24.779285 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:24.779393 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:38429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":311,"threads_started":1}
I20260926 16:35:25.232869 23352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:25.231878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:38429 (local address 127.6.33.2:49041)
0926 16:35:25.232465 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:25.232567 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:38429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":286,"threads_started":1}
I20260926 16:35:25.703439 23353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:25.702537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:38429 (local address 127.6.33.3:54977)
0926 16:35:25.703154 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:25.703248 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:38429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":270,"threads_started":1}
I20260926 16:35:25.781634 23354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:25.780839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:38429 (local address 127.6.33.1:41533)
0926 16:35:25.781371 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:25.781459 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:38429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":301,"threads_started":1}
I20260926 16:35:26.234895 23355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:26.233959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:38429 (local address 127.6.33.2:45239)
0926 16:35:26.234434 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:26.234700 (+ 266us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:38429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":269,"threads_started":1}
I20260926 16:35:26.705222 23356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:26.704453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:38429 (local address 127.6.33.3:38971)
0926 16:35:26.704944 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:26.705025 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:38429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":237,"threads_started":1}
I20260926 16:35:26.776952 23357 raft_consensus.cc:482] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P f1f4f18f75344a569ccd2bf27a10f8d7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:35:26.777199 23357 raft_consensus.cc:504] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P f1f4f18f75344a569ccd2bf27a10f8d7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5eb6546b9642e5b64b22da3a5298ca" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 32913 } } peers { permanent_uuid: "6a190cafd1f94f38ad1c6bf176f8066b" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45075 } } peers { permanent_uuid: "f1f4f18f75344a569ccd2bf27a10f8d7" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 41569 } }
I20260926 16:35:26.777824 23357 leader_election.cc:288] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P f1f4f18f75344a569ccd2bf27a10f8d7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fa5eb6546b9642e5b64b22da3a5298ca (127.6.33.2:32913), 6a190cafd1f94f38ad1c6bf176f8066b (127.6.33.3:45075)
W20260926 16:35:26.778976 22949 leader_election.cc:341] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P f1f4f18f75344a569ccd2bf27a10f8d7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fa5eb6546b9642e5b64b22da3a5298ca (127.6.33.2:32913): Not found: Tablet not found: 84928aa4779f4f1ea6e32a26c4b3d8b1
W20260926 16:35:26.779250 22949 leader_election.cc:341] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P f1f4f18f75344a569ccd2bf27a10f8d7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6a190cafd1f94f38ad1c6bf176f8066b (127.6.33.3:45075): Not found: Tablet not found: 84928aa4779f4f1ea6e32a26c4b3d8b1
I20260926 16:35:26.779361 22949 leader_election.cc:302] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P f1f4f18f75344a569ccd2bf27a10f8d7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1f4f18f75344a569ccd2bf27a10f8d7; no voters: 6a190cafd1f94f38ad1c6bf176f8066b, fa5eb6546b9642e5b64b22da3a5298ca
I20260926 16:35:26.779686 23357 raft_consensus.cc:2726] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P f1f4f18f75344a569ccd2bf27a10f8d7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 16:35:26.783339 23358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:26.782553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:38429 (local address 127.6.33.1:51705)
0926 16:35:26.783023 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:26.783103 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:38429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":243,"threads_started":1}
I20260926 16:35:27.236613 23359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:27.235849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:38429 (local address 127.6.33.2:36361)
0926 16:35:27.236359 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:27.236450 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:38429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":237,"threads_started":1}
I20260926 16:35:27.707443 23360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:27.706338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:38429 (local address 127.6.33.3:55261)
0926 16:35:27.707104 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:27.707221 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:38429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":281,"threads_started":1}
I20260926 16:35:27.785604 23361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:27.784678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:38429 (local address 127.6.33.1:57129)
0926 16:35:27.785324 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:27.785416 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:38429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":413,"threads_started":1}
I20260926 16:35:28.238668 23362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:28.237786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:38429 (local address 127.6.33.2:52799)
0926 16:35:28.238377 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:28.238466 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:38429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":352,"threads_started":1}
I20260926 16:35:28.709503 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:28.708661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:38429 (local address 127.6.33.3:36209)
0926 16:35:28.709224 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:28.709316 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:38429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":312,"threads_started":1}
I20260926 16:35:28.787356 23364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:28.786625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:38429 (local address 127.6.33.1:57251)
0926 16:35:28.787085 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:28.787179 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:38429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":252,"threads_started":1}
I20260926 16:35:29.240715 23365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:29.239807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:38429 (local address 127.6.33.2:42883)
0926 16:35:29.240430 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:29.240532 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:38429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":321,"threads_started":1}
I20260926 16:35:29.711409 23366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:29.710520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:38429 (local address 127.6.33.3:38449)
0926 16:35:29.711166 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:29.711250 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:38429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":390,"threads_started":1}
I20260926 16:35:29.789393 23367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:29.788451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:38429 (local address 127.6.33.1:50073)
0926 16:35:29.789121 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:29.789230 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:38429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":332,"threads_started":1}
I20260926 16:35:29.806205 23368 raft_consensus.cc:482] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P f1f4f18f75344a569ccd2bf27a10f8d7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:35:29.806433 23368 raft_consensus.cc:504] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P f1f4f18f75344a569ccd2bf27a10f8d7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5eb6546b9642e5b64b22da3a5298ca" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 32913 } } peers { permanent_uuid: "6a190cafd1f94f38ad1c6bf176f8066b" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45075 } } peers { permanent_uuid: "f1f4f18f75344a569ccd2bf27a10f8d7" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 41569 } }
I20260926 16:35:29.807173 23368 leader_election.cc:288] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P f1f4f18f75344a569ccd2bf27a10f8d7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fa5eb6546b9642e5b64b22da3a5298ca (127.6.33.2:32913), 6a190cafd1f94f38ad1c6bf176f8066b (127.6.33.3:45075)
W20260926 16:35:29.808144 22949 leader_election.cc:341] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P f1f4f18f75344a569ccd2bf27a10f8d7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fa5eb6546b9642e5b64b22da3a5298ca (127.6.33.2:32913): Not found: Tablet not found: 84928aa4779f4f1ea6e32a26c4b3d8b1
W20260926 16:35:29.808363 22949 leader_election.cc:341] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P f1f4f18f75344a569ccd2bf27a10f8d7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6a190cafd1f94f38ad1c6bf176f8066b (127.6.33.3:45075): Not found: Tablet not found: 84928aa4779f4f1ea6e32a26c4b3d8b1
I20260926 16:35:29.808486 22949 leader_election.cc:302] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P f1f4f18f75344a569ccd2bf27a10f8d7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1f4f18f75344a569ccd2bf27a10f8d7; no voters: 6a190cafd1f94f38ad1c6bf176f8066b, fa5eb6546b9642e5b64b22da3a5298ca
I20260926 16:35:29.808804 23368 raft_consensus.cc:2726] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P f1f4f18f75344a569ccd2bf27a10f8d7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 16:35:30.242451 23369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:30.241693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:38429 (local address 127.6.33.2:39511)
0926 16:35:30.242174 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:30.242262 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:38429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":244,"threads_started":1}
I20260926 16:35:30.713582 23370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:30.712669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:38429 (local address 127.6.33.3:39883)
0926 16:35:30.713288 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:30.713394 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:38429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":303,"threads_started":1}
I20260926 16:35:30.791337 23371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:30.790523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:38429 (local address 127.6.33.1:35773)
0926 16:35:30.791044 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:30.791153 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:38429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":257,"threads_started":1}
I20260926 16:35:31.244571 23372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:31.243564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:38429 (local address 127.6.33.2:45619)
0926 16:35:31.244249 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:31.244382 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:38429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":269,"threads_started":1}
I20260926 16:35:31.715608 23373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:31.714809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:38429 (local address 127.6.33.3:35205)
0926 16:35:31.715342 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:31.715437 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:38429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":272,"threads_started":1}
I20260926 16:35:31.793387 23374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:31.792548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:38429 (local address 127.6.33.1:39991)
0926 16:35:31.793091 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:31.793184 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:38429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":312,"threads_started":1}
I20260926 16:35:32.246647 23375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:32.245753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:38429 (local address 127.6.33.2:50671)
0926 16:35:32.246259 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:32.246348 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:38429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":247,"threads_started":1}
I20260926 16:35:32.564846 23376 raft_consensus.cc:482] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P f1f4f18f75344a569ccd2bf27a10f8d7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:35:32.565037 23376 raft_consensus.cc:504] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P f1f4f18f75344a569ccd2bf27a10f8d7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5eb6546b9642e5b64b22da3a5298ca" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 32913 } } peers { permanent_uuid: "6a190cafd1f94f38ad1c6bf176f8066b" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45075 } } peers { permanent_uuid: "f1f4f18f75344a569ccd2bf27a10f8d7" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 41569 } }
I20260926 16:35:32.565605 23376 leader_election.cc:288] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P f1f4f18f75344a569ccd2bf27a10f8d7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fa5eb6546b9642e5b64b22da3a5298ca (127.6.33.2:32913), 6a190cafd1f94f38ad1c6bf176f8066b (127.6.33.3:45075)
W20260926 16:35:32.566716 22949 leader_election.cc:341] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P f1f4f18f75344a569ccd2bf27a10f8d7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6a190cafd1f94f38ad1c6bf176f8066b (127.6.33.3:45075): Not found: Tablet not found: 84928aa4779f4f1ea6e32a26c4b3d8b1
W20260926 16:35:32.566969 22949 leader_election.cc:341] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P f1f4f18f75344a569ccd2bf27a10f8d7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fa5eb6546b9642e5b64b22da3a5298ca (127.6.33.2:32913): Not found: Tablet not found: 84928aa4779f4f1ea6e32a26c4b3d8b1
I20260926 16:35:32.567085 22949 leader_election.cc:302] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P f1f4f18f75344a569ccd2bf27a10f8d7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1f4f18f75344a569ccd2bf27a10f8d7; no voters: 6a190cafd1f94f38ad1c6bf176f8066b, fa5eb6546b9642e5b64b22da3a5298ca
I20260926 16:35:32.567417 23376 raft_consensus.cc:2726] T 84928aa4779f4f1ea6e32a26c4b3d8b1 P f1f4f18f75344a569ccd2bf27a10f8d7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 16:35:32.717598 23377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:32.716752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:38429 (local address 127.6.33.3:60489)
0926 16:35:32.717287 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:32.717376 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:38429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":259,"threads_started":1}
I20260926 16:35:32.795569 23378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:32.794730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:38429 (local address 127.6.33.1:51665)
0926 16:35:32.795273 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:32.795365 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:38429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":281,"threads_started":1}
I20260926 16:35:33.248544 23379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:33.247837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:38429 (local address 127.6.33.2:60931)
0926 16:35:33.248289 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:33.248371 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:38429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":217,"threads_started":1}
I20260926 16:35:33.694999 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 22890
I20260926 16:35:33.711024 23380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:33.696311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53566 (local address 127.6.33.1:41569)
0926 16:35:33.697090 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:33.697101 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:33.697139 (+ 38us) server_negotiation.cc:407] Connection header received
0926 16:35:33.697234 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:33.697246 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:33.697569 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:33.697782 (+ 213us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:35:33.699134 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:33.700138 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:33.701486 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:33.702056 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:33.702166 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:33.704680 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:35:33.704715 (+ 35us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:35:33.704733 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:35:33.704833 (+ 100us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:35:33.707962 (+ 3129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:35:33.708842 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:35:33.708850 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:35:33.708852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:35:33.709012 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:35:33.709375 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:35:33.709378 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:35:33.709380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:35:33.709755 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:35:33.709999 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:33.710329 (+ 330us) server_negotiation.cc:299] Negotiation successful
0926 16:35:33.710569 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":373,"threads_started":1}
I20260926 16:35:33.712453 23055 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:35:33.719769 23382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:33.718948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:38429 (local address 127.6.33.3:46329)
0926 16:35:33.719478 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:33.719581 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.62:38429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":304,"threads_started":1}
I20260926 16:35:33.804625 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 22890
I20260926 16:35:33.822639 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 23063
I20260926 16:35:33.836931 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:33.823598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54728 (local address 127.6.33.2:32913)
0926 16:35:33.824107 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:33.824112 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:33.824127 (+ 15us) server_negotiation.cc:407] Connection header received
0926 16:35:33.824182 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:33.824186 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:33.824244 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:33.824364 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:35:33.825819 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:33.826657 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:33.827935 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:33.828391 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:33.828522 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:33.831487 (+ 2965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:35:33.831494 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:35:33.831499 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:35:33.831540 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:35:33.834461 (+ 2921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:35:33.835117 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:35:33.835121 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:35:33.835123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:35:33.835269 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:35:33.835639 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:35:33.835642 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:35:33.835644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:35:33.835917 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:35:33.836123 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:33.836355 (+ 232us) server_negotiation.cc:299] Negotiation successful
0926 16:35:33.836534 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":272,"threads_started":1}
I20260926 16:35:33.837659 23187 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:35:33.922885 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 23063
I20260926 16:35:33.940678 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 23195
I20260926 16:35:33.955505 23386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:33.941750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59772 (local address 127.6.33.3:45075)
0926 16:35:33.942295 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:33.942300 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:33.942317 (+ 17us) server_negotiation.cc:407] Connection header received
0926 16:35:33.942368 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:33.942372 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:33.942436 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:33.942557 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:35:33.944009 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:33.944854 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:33.946069 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:33.946524 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:33.946709 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:33.949576 (+ 2867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:35:33.949582 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:35:33.949585 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:35:33.949618 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:35:33.952235 (+ 2617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:35:33.953038 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:35:33.953047 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:35:33.953050 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:35:33.953208 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:35:33.953633 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:35:33.953638 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:35:33.953641 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:35:33.954060 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:35:33.954334 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:33.954807 (+ 473us) server_negotiation.cc:299] Negotiation successful
0926 16:35:33.955066 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":257,"threads_started":1}
I20260926 16:35:33.956292 23318 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:35:34.041440 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 23195
2026-09-26T16:35:34Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64708 ms)
[----------] 30 tests from SecurityITest (300157 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:35:34 dist-test-slave-khd3 krb5kdc[23390](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:35:34 dist-test-slave-khd3 krb5kdc[23390](info): set up 2 sockets
Sep 26 16:35:34 dist-test-slave-khd3 krb5kdc[23390](info): commencing operation
krb5kdc: starting...
W20260926 16:35:35.189869 6276 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Sep 26 16:35:35 dist-test-slave-khd3 krb5kdc[23390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:35:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:35:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab.
Entry for principal kudu/127.6.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab.
I20260926 16:35:35.445477 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:43779
--webserver_interface=127.6.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:45989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.33.62:43779
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:35:35.793006 23403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:35:35.793366 23403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:35:35.793525 23403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:35:35.800945 23403 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:35:35.801088 23403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:35:35.801195 23403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:35:35.801301 23403 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:35:35.801388 23403 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:35:35.811663 23403 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:45989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.33.62:43779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:43779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:35:35.813118 23403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:35:35.814555 23403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:35:35.827062 23409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:35:35.827160 23408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:35:35.827167 23411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:35:35.827806 23403 server_base.cc:1034] running on GCE node
I20260926 16:35:35.828635 23403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:35:35.830291 23403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:35:35.831524 23403 hybrid_clock.cc:648] HybridClock initialized: now 1790440535831438 us; error 67 us; skew 500 ppm
I20260926 16:35:35.831853 23403 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:35:35 dist-test-slave-khd3 krb5kdc[23390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440535, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:35:35.839418 23403 init.cc:377] Logged in from keytab as kudu/127.6.33.62@KRBTEST.COM (short username kudu)
I20260926 16:35:35.841200 23403 webserver.cc:460] Webserver started at http://127.6.33.62:43281/ using document root <none> and password file <none>
I20260926 16:35:35.841940 23403 fs_manager.cc:359] Metadata directory not provided
I20260926 16:35:35.842046 23403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:35:35.842314 23403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:35:35.845566 23403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/master-0/data/instance:
uuid: "71ff3629338140bfb9414b7ff046ec9a"
format_stamp: "Formatted at 2026-09-26 16:35:35 on dist-test-slave-khd3"
server_key: "6016e77cf0b79a2a0006ae98d2146990"
server_key_iv: "6b5edf170649a954a76104d48c2516b1"
server_key_version: "encryptionkey@0"
I20260926 16:35:35.846354 23403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/master-0/wal/instance:
uuid: "71ff3629338140bfb9414b7ff046ec9a"
format_stamp: "Formatted at 2026-09-26 16:35:35 on dist-test-slave-khd3"
server_key: "6016e77cf0b79a2a0006ae98d2146990"
server_key_iv: "6b5edf170649a954a76104d48c2516b1"
server_key_version: "encryptionkey@0"
I20260926 16:35:35.852662 23403 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.003s
I20260926 16:35:35.856962 23418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:35:35.858543 23403 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:35:35.858771 23403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/master-0/wal
uuid: "71ff3629338140bfb9414b7ff046ec9a"
format_stamp: "Formatted at 2026-09-26 16:35:35 on dist-test-slave-khd3"
server_key: "6016e77cf0b79a2a0006ae98d2146990"
server_key_iv: "6b5edf170649a954a76104d48c2516b1"
server_key_version: "encryptionkey@0"
I20260926 16:35:35.859100 23403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:35:35.875185 23403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:35:35.880132 23403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:35:35.880455 23403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:35:35.900398 23403 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.62:43779
I20260926 16:35:35.900440 23469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.62:43779 every 8 connection(s)
I20260926 16:35:35.901938 23403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
I20260926 16:35:35.906103 23470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:35:35.910526 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 23403
I20260926 16:35:35.910782 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/master-0/wal/instance
I20260926 16:35:35.911636 6276 external_mini_cluster.cc:1428] Setting key 4a3ccd56da9db0002a2c84b2f83e43ba
I20260926 16:35:35.920415 23470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71ff3629338140bfb9414b7ff046ec9a: Bootstrap starting.
I20260926 16:35:35.923834 23470 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71ff3629338140bfb9414b7ff046ec9a: Neither blocks nor log segments found. Creating new log.
I20260926 16:35:35.925061 23470 log.cc:824] T 00000000000000000000000000000000 P 71ff3629338140bfb9414b7ff046ec9a: Log is configured to *not* fsync() on all Append() calls
Sep 26 16:35:35 dist-test-slave-khd3 krb5kdc[23390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:35:35.928323 23470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71ff3629338140bfb9414b7ff046ec9a: No bootstrap required, opened a new log
I20260926 16:35:35.935817 23473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:35.914183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42442 (local address 127.6.33.62:43779)
0926 16:35:35.914974 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:35.914987 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:35.915036 (+ 49us) server_negotiation.cc:407] Connection header received
0926 16:35:35.916730 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:35.916785 (+ 55us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:35.917269 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:35.917727 (+ 458us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:35:35.919261 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:35.920660 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:35.922079 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:35.922785 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:35.922993 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:35.927713 (+ 4720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:35:35.927722 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:35:35.927738 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:35:35.927777 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:35:35.931276 (+ 3499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:35:35.932136 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:35:35.932148 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:35:35.932156 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:35:35.932325 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:35:35.932749 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:35:35.932791 (+ 42us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:35:35.932795 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:35:35.933406 (+ 611us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:35:35.933737 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:35.934242 (+ 505us) server_negotiation.cc:299] Negotiation successful
0926 16:35:35.934710 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":330,"threads_started":1}
I20260926 16:35:35.936587 23470 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71ff3629338140bfb9414b7ff046ec9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71ff3629338140bfb9414b7ff046ec9a" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 43779 } }
I20260926 16:35:35.937119 23470 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71ff3629338140bfb9414b7ff046ec9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:35:35.937242 23470 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71ff3629338140bfb9414b7ff046ec9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71ff3629338140bfb9414b7ff046ec9a, State: Initialized, Role: FOLLOWER
I20260926 16:35:35.937925 23470 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71ff3629338140bfb9414b7ff046ec9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71ff3629338140bfb9414b7ff046ec9a" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 43779 } }
I20260926 16:35:35.938184 23470 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71ff3629338140bfb9414b7ff046ec9a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:35:35.938316 23470 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71ff3629338140bfb9414b7ff046ec9a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:35:35.938498 23470 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71ff3629338140bfb9414b7ff046ec9a [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:35:35.941015 23470 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71ff3629338140bfb9414b7ff046ec9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71ff3629338140bfb9414b7ff046ec9a" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 43779 } }
I20260926 16:35:35.941579 23470 leader_election.cc:302] T 00000000000000000000000000000000 P 71ff3629338140bfb9414b7ff046ec9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71ff3629338140bfb9414b7ff046ec9a; no voters:
I20260926 16:35:35.942195 23470 leader_election.cc:288] T 00000000000000000000000000000000 P 71ff3629338140bfb9414b7ff046ec9a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:35:35.942374 23475 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71ff3629338140bfb9414b7ff046ec9a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:35:35.943125 23475 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71ff3629338140bfb9414b7ff046ec9a [term 1 LEADER]: Becoming Leader. State: Replica: 71ff3629338140bfb9414b7ff046ec9a, State: Running, Role: LEADER
I20260926 16:35:35.943698 23475 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71ff3629338140bfb9414b7ff046ec9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71ff3629338140bfb9414b7ff046ec9a" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 43779 } }
I20260926 16:35:35.944450 23470 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71ff3629338140bfb9414b7ff046ec9a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:35:35.946978 23477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71ff3629338140bfb9414b7ff046ec9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71ff3629338140bfb9414b7ff046ec9a. Latest consensus state: current_term: 1 leader_uuid: "71ff3629338140bfb9414b7ff046ec9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71ff3629338140bfb9414b7ff046ec9a" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 43779 } } }
I20260926 16:35:35.947486 23477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71ff3629338140bfb9414b7ff046ec9a [sys.catalog]: This master's current role is: LEADER
I20260926 16:35:35.947104 23476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71ff3629338140bfb9414b7ff046ec9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71ff3629338140bfb9414b7ff046ec9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71ff3629338140bfb9414b7ff046ec9a" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 43779 } } }
I20260926 16:35:35.947762 23476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71ff3629338140bfb9414b7ff046ec9a [sys.catalog]: This master's current role is: LEADER
I20260926 16:35:35.949533 23482 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:35:35.955407 23482 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:35:35.965263 23482 catalog_manager.cc:1347] Generated new cluster ID: 1eed945f0f2248ff98f95af0fbf0cf88
I20260926 16:35:35.965384 23482 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:35:35.989959 23482 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:35:35.991434 23482 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:35:35.999398 23482 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71ff3629338140bfb9414b7ff046ec9a: Generated new TSK 0
I20260926 16:35:36.000296 23482 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.33.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab.
Entry for principal kudu/127.6.33.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab.
I20260926 16:35:36.065052 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.1:0
--local_ip_for_outbound_sockets=127.6.33.1
--webserver_interface=127.6.33.1
--webserver_port=0
--tserver_master_addrs=127.6.33.62:43779
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab
--principal=kudu/127.6.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:45989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:35:36.407183 23496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:35:36.407476 23496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:35:36.407662 23496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:35:36.415011 23496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:35:36.415128 23496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:35:36.415246 23496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.1
I20260926 16:35:36.425562 23496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:45989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab
--principal=kudu/127.6.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.33.1
--webserver_port=0
--tserver_master_addrs=127.6.33.62:43779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.1
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:35:36.426829 23496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:35:36.428277 23496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:35:36.442884 23504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:35:36.442931 23502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:35:36.442957 23501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:35:36.443944 23496 server_base.cc:1034] running on GCE node
I20260926 16:35:36.444861 23496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:35:36.446046 23496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:35:36.447278 23496 hybrid_clock.cc:648] HybridClock initialized: now 1790440536447231 us; error 49 us; skew 500 ppm
I20260926 16:35:36.447603 23496 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:35:36 dist-test-slave-khd3 krb5kdc[23390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440536, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:35:36.451409 23496 init.cc:377] Logged in from keytab as kudu/127.6.33.1@KRBTEST.COM (short username kudu)
I20260926 16:35:36.453087 23496 webserver.cc:460] Webserver started at http://127.6.33.1:37801/ using document root <none> and password file <none>
I20260926 16:35:36.453841 23496 fs_manager.cc:359] Metadata directory not provided
I20260926 16:35:36.453950 23496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:35:36.454275 23496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:35:36.457559 23496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/ts-0/data/instance:
uuid: "cd25c2b8cb904630b400168f25ef0db8"
format_stamp: "Formatted at 2026-09-26 16:35:36 on dist-test-slave-khd3"
server_key: "47a7650c3819e11162ae8a80d61dbfc7"
server_key_iv: "0b75d798e900d0af01cdc894e45f80c6"
server_key_version: "encryptionkey@0"
I20260926 16:35:36.458391 23496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/ts-0/wal/instance:
uuid: "cd25c2b8cb904630b400168f25ef0db8"
format_stamp: "Formatted at 2026-09-26 16:35:36 on dist-test-slave-khd3"
server_key: "47a7650c3819e11162ae8a80d61dbfc7"
server_key_iv: "0b75d798e900d0af01cdc894e45f80c6"
server_key_version: "encryptionkey@0"
I20260926 16:35:36.464424 23496 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260926 16:35:36.468205 23511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:35:36.469764 23496 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:35:36.469946 23496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/ts-0/wal
uuid: "cd25c2b8cb904630b400168f25ef0db8"
format_stamp: "Formatted at 2026-09-26 16:35:36 on dist-test-slave-khd3"
server_key: "47a7650c3819e11162ae8a80d61dbfc7"
server_key_iv: "0b75d798e900d0af01cdc894e45f80c6"
server_key_version: "encryptionkey@0"
I20260926 16:35:36.470150 23496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:35:36.481940 23496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:35:36.486552 23496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:35:36.486865 23496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:35:36.487884 23496 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:35:36.489355 23496 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:35:36.489425 23496 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:35:36.489566 23496 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:35:36.489625 23496 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:35:36.522843 23496 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.1:44979
I20260926 16:35:36.522892 23623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.1:44979 every 8 connection(s)
I20260926 16:35:36.524224 23496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
I20260926 16:35:36.530256 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 23496
I20260926 16:35:36.530486 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/ts-0/wal/instance
I20260926 16:35:36.531204 6276 external_mini_cluster.cc:1428] Setting key 6d8d4f261233cb3b4884a0aafc3795ed
Sep 26 16:35:36 dist-test-slave-khd3 krb5kdc[23390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440536, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.1@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:35:36.544895 23626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:36.526931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:51939 (local address 127.6.33.62:43779)
0926 16:35:36.527396 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:36.527402 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:36.528422 (+ 1020us) server_negotiation.cc:407] Connection header received
0926 16:35:36.530321 (+ 1899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:36.530326 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:36.530428 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:36.530564 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:35:36.532290 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:36.533231 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:36.534829 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:36.535254 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:36.535389 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:36.539029 (+ 3640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:35:36.539034 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:35:36.539038 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:35:36.539071 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:35:36.541783 (+ 2712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:35:36.542527 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:35:36.542532 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:35:36.542538 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:35:36.542661 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:35:36.543078 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:35:36.543086 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:35:36.543089 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:35:36.543367 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:35:36.543524 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:36.544267 (+ 743us) server_negotiation.cc:299] Negotiation successful
0926 16:35:36.544504 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":211,"threads_started":1}
I20260926 16:35:36.545996 23627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:36.527546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:43779 (local address 127.6.33.1:51939)
0926 16:35:36.528255 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:36.528300 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 16:35:36.530033 (+ 1733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:35:36.530817 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:35:36.530826 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:35:36.531409 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:35:36.532120 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:36.532141 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:36.533400 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:36.533405 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:36.534686 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:36.534698 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:36.535487 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:36.535494 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:36.535746 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:36.536471 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:35:36.536494 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:35:36.538791 (+ 2297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:35:36.541936 (+ 3145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:35:36.541945 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:35:36.541963 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:35:36.542405 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:35:36.542771 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:35:36.542776 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:35:36.542780 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:35:36.542967 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:35:36.543498 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:35:36.543507 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:35:36.543867 (+ 360us) client_negotiation.cc:769] Sending connection context
0926 16:35:36.544163 (+ 296us) client_negotiation.cc:240] Negotiation successful
0926 16:35:36.544775 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":278,"threads_started":1}
I20260926 16:35:36.547664 23624 heartbeater.cc:347] Connected to a master server at 127.6.33.62:43779
I20260926 16:35:36.548012 23624 heartbeater.cc:464] Registering TS with master...
I20260926 16:35:36.548978 23624 heartbeater.cc:511] Master 127.6.33.62:43779 requested a full tablet report, sending...
I20260926 16:35:36.551582 23435 ts_manager.cc:194] Registered new tserver with Master: cd25c2b8cb904630b400168f25ef0db8 (127.6.33.1:44979)
I20260926 16:35:36.553984 23435 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.33.1@KRBTEST.COM'} at 127.6.33.1:51939
WARNING: no policy specified for kudu/127.6.33.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab.
Entry for principal kudu/127.6.33.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab.
I20260926 16:35:36.588588 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.2:0
--local_ip_for_outbound_sockets=127.6.33.2
--webserver_interface=127.6.33.2
--webserver_port=0
--tserver_master_addrs=127.6.33.62:43779
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab
--principal=kudu/127.6.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:45989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:35:36.929898 23630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:35:36.930208 23630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:35:36.930364 23630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:35:36.937935 23630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:35:36.938058 23630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:35:36.938277 23630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.2
I20260926 16:35:36.948966 23630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:45989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab
--principal=kudu/127.6.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.33.2
--webserver_port=0
--tserver_master_addrs=127.6.33.62:43779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.2
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:35:36.950454 23630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:35:36.951908 23630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:35:36.965870 23635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:35:36.965957 23636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:35:36.966280 23638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:35:36.967450 23630 server_base.cc:1034] running on GCE node
I20260926 16:35:36.968144 23630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:35:36.969295 23630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:35:36.970614 23630 hybrid_clock.cc:648] HybridClock initialized: now 1790440536970548 us; error 60 us; skew 500 ppm
I20260926 16:35:36.971006 23630 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:35:36 dist-test-slave-khd3 krb5kdc[23390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440536, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:35:36.974628 23630 init.cc:377] Logged in from keytab as kudu/127.6.33.2@KRBTEST.COM (short username kudu)
I20260926 16:35:36.976400 23630 webserver.cc:460] Webserver started at http://127.6.33.2:43831/ using document root <none> and password file <none>
I20260926 16:35:36.977281 23630 fs_manager.cc:359] Metadata directory not provided
I20260926 16:35:36.977439 23630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:35:36.977957 23630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:35:36.981223 23630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/ts-1/data/instance:
uuid: "1cdf861ba35547568ba6bda82753dfac"
format_stamp: "Formatted at 2026-09-26 16:35:36 on dist-test-slave-khd3"
server_key: "011680fc2f09f2b2abb1e712da4b8b95"
server_key_iv: "f9cb4a8e8bf643a753bb3b9901e4a33e"
server_key_version: "encryptionkey@0"
I20260926 16:35:36.982115 23630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/ts-1/wal/instance:
uuid: "1cdf861ba35547568ba6bda82753dfac"
format_stamp: "Formatted at 2026-09-26 16:35:36 on dist-test-slave-khd3"
server_key: "011680fc2f09f2b2abb1e712da4b8b95"
server_key_iv: "f9cb4a8e8bf643a753bb3b9901e4a33e"
server_key_version: "encryptionkey@0"
I20260926 16:35:36.988154 23630 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 16:35:36.992308 23645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:35:36.993873 23630 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:35:36.994082 23630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/ts-1/wal
uuid: "1cdf861ba35547568ba6bda82753dfac"
format_stamp: "Formatted at 2026-09-26 16:35:36 on dist-test-slave-khd3"
server_key: "011680fc2f09f2b2abb1e712da4b8b95"
server_key_iv: "f9cb4a8e8bf643a753bb3b9901e4a33e"
server_key_version: "encryptionkey@0"
I20260926 16:35:36.994313 23630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:35:37.006749 23630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:35:37.011221 23630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:35:37.011554 23630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:35:37.012689 23630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:35:37.014179 23630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:35:37.014289 23630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:35:37.014449 23630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:35:37.014524 23630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:35:37.051614 23630 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.2:45723
I20260926 16:35:37.051710 23757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.2:45723 every 8 connection(s)
I20260926 16:35:37.053089 23630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
I20260926 16:35:37.053504 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 23630
I20260926 16:35:37.053720 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/ts-1/wal/instance
I20260926 16:35:37.054307 6276 external_mini_cluster.cc:1428] Setting key 2b3caad60523d898819bcd38f061a1bf
Sep 26 16:35:37 dist-test-slave-khd3 krb5kdc[23390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440536, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.2@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:35:37.077001 23760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:37.056556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:58919 (local address 127.6.33.62:43779)
0926 16:35:37.057085 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:37.057090 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:37.058430 (+ 1340us) server_negotiation.cc:407] Connection header received
0926 16:35:37.060505 (+ 2075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:37.060511 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:37.060623 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:37.060749 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:35:37.062723 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:37.063753 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:37.065666 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:37.066177 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:37.066334 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:37.070341 (+ 4007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:35:37.070347 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:35:37.070350 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:35:37.070388 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:35:37.073402 (+ 3014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:35:37.074381 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:35:37.074385 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:35:37.074388 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:35:37.074535 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:35:37.075011 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:35:37.075023 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:35:37.075026 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:35:37.075374 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:35:37.075545 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:37.076303 (+ 758us) server_negotiation.cc:299] Negotiation successful
0926 16:35:37.076555 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":4239,"threads_started":1}
I20260926 16:35:37.078071 23761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:37.057372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:43779 (local address 127.6.33.2:58919)
0926 16:35:37.058264 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:37.058307 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 16:35:37.060154 (+ 1847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:35:37.061045 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:35:37.061054 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:35:37.061634 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:35:37.062453 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:37.062477 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:37.063928 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:37.063940 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:37.065481 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:37.065497 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:37.066456 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:37.066463 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:37.066725 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:37.067523 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:35:37.067565 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:35:37.070033 (+ 2468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:35:37.073600 (+ 3567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:35:37.073616 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:35:37.073631 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:35:37.074221 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:35:37.074676 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:35:37.074680 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:35:37.074682 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:35:37.074884 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:35:37.075517 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:35:37.075525 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:35:37.075920 (+ 395us) client_negotiation.cc:769] Sending connection context
0926 16:35:37.076208 (+ 288us) client_negotiation.cc:240] Negotiation successful
0926 16:35:37.076860 (+ 652us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":360,"threads_started":1}
WARNING: no policy specified for kudu/127.6.33.3@KRBTEST.COM; defaulting to no policy
I20260926 16:35:37.079797 23758 heartbeater.cc:347] Connected to a master server at 127.6.33.62:43779
I20260926 16:35:37.080195 23758 heartbeater.cc:464] Registering TS with master...
I20260926 16:35:37.081116 23758 heartbeater.cc:511] Master 127.6.33.62:43779 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.3@KRBTEST.COM" created.
I20260926 16:35:37.082998 23435 ts_manager.cc:194] Registered new tserver with Master: 1cdf861ba35547568ba6bda82753dfac (127.6.33.2:45723)
I20260926 16:35:37.083916 23435 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.33.2@KRBTEST.COM'} at 127.6.33.2:58919
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab.
Entry for principal kudu/127.6.33.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab.
I20260926 16:35:37.113137 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.3:0
--local_ip_for_outbound_sockets=127.6.33.3
--webserver_interface=127.6.33.3
--webserver_port=0
--tserver_master_addrs=127.6.33.62:43779
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab
--principal=kudu/127.6.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:45989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:35:37.459949 23764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:35:37.460275 23764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:35:37.460448 23764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:35:37.468294 23764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:35:37.468417 23764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:35:37.468608 23764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.3
I20260926 16:35:37.479218 23764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:45989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab
--principal=kudu/127.6.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.33.3
--webserver_port=0
--tserver_master_addrs=127.6.33.62:43779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.3
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:35:37.480579 23764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:35:37.482018 23764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:35:37.495266 23769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:35:37.495421 23772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:35:37.495402 23770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:35:37.497454 23764 server_base.cc:1034] running on GCE node
I20260926 16:35:37.498189 23764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:35:37.499334 23764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:35:37.500620 23764 hybrid_clock.cc:648] HybridClock initialized: now 1790440537500541 us; error 61 us; skew 500 ppm
I20260926 16:35:37.501300 23764 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:35:37 dist-test-slave-khd3 krb5kdc[23390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440537, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:35:37.505029 23764 init.cc:377] Logged in from keytab as kudu/127.6.33.3@KRBTEST.COM (short username kudu)
I20260926 16:35:37.506820 23764 webserver.cc:460] Webserver started at http://127.6.33.3:38873/ using document root <none> and password file <none>
I20260926 16:35:37.507658 23764 fs_manager.cc:359] Metadata directory not provided
I20260926 16:35:37.507815 23764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:35:37.508152 23764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:35:37.511325 23764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/ts-2/data/instance:
uuid: "b7e1b2c89bfb4344b9c72054f28c5ac8"
format_stamp: "Formatted at 2026-09-26 16:35:37 on dist-test-slave-khd3"
server_key: "9be90dc6ba861c5b52deaa77f4e96e74"
server_key_iv: "93d257c00a2a9277839ee07a7f531405"
server_key_version: "encryptionkey@0"
I20260926 16:35:37.512172 23764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/ts-2/wal/instance:
uuid: "b7e1b2c89bfb4344b9c72054f28c5ac8"
format_stamp: "Formatted at 2026-09-26 16:35:37 on dist-test-slave-khd3"
server_key: "9be90dc6ba861c5b52deaa77f4e96e74"
server_key_iv: "93d257c00a2a9277839ee07a7f531405"
server_key_version: "encryptionkey@0"
I20260926 16:35:37.517804 23764 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 16:35:37.521791 23779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:35:37.523573 23764 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 16:35:37.523761 23764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/ts-2/wal
uuid: "b7e1b2c89bfb4344b9c72054f28c5ac8"
format_stamp: "Formatted at 2026-09-26 16:35:37 on dist-test-slave-khd3"
server_key: "9be90dc6ba861c5b52deaa77f4e96e74"
server_key_iv: "93d257c00a2a9277839ee07a7f531405"
server_key_version: "encryptionkey@0"
I20260926 16:35:37.523988 23764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:35:37.540481 23764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:35:37.544869 23764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:35:37.545166 23764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:35:37.546219 23764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:35:37.547631 23764 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:35:37.547714 23764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:35:37.547832 23764 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:35:37.547899 23764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:35:37.558115 23624 heartbeater.cc:503] Master 127.6.33.62:43779 was elected leader, sending a full tablet report...
I20260926 16:35:37.582823 23764 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.3:41267
I20260926 16:35:37.582872 23891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.3:41267 every 8 connection(s)
I20260926 16:35:37.584208 23764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
I20260926 16:35:37.588961 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 23764
I20260926 16:35:37.589143 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/ts-2/wal/instance
I20260926 16:35:37.589763 6276 external_mini_cluster.cc:1428] Setting key b1c327ec90ac367178f4805ddec3445e
Sep 26 16:35:37 dist-test-slave-khd3 krb5kdc[23390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440537, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.3@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:35:37.605835 23894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:37.586802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.3:58217 (local address 127.6.33.62:43779)
0926 16:35:37.587257 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:37.587261 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:37.588148 (+ 887us) server_negotiation.cc:407] Connection header received
0926 16:35:37.590703 (+ 2555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:37.590708 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:37.590823 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:37.590939 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:35:37.592634 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:37.593489 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:37.595071 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:37.595474 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:37.595642 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:37.599239 (+ 3597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:35:37.599244 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:35:37.599247 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:35:37.599281 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:35:37.602028 (+ 2747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:35:37.602822 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:35:37.602828 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:35:37.602832 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:35:37.603006 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:35:37.603815 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:35:37.603818 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:35:37.603821 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:35:37.604204 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:35:37.604374 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:37.605331 (+ 957us) server_negotiation.cc:299] Negotiation successful
0926 16:35:37.605550 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":238,"threads_started":1}
I20260926 16:35:37.606850 23895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:37.587248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:43779 (local address 127.6.33.3:58217)
0926 16:35:37.587960 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:37.588002 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 16:35:37.590408 (+ 2406us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:35:37.591237 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:35:37.591249 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:35:37.591870 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:35:37.592455 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:37.592472 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:37.593644 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:37.593652 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:37.594914 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:37.594925 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:37.595723 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:37.595730 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:37.595964 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:37.596655 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:35:37.596678 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:35:37.599007 (+ 2329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:35:37.602210 (+ 3203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:35:37.602219 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:35:37.602238 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:35:37.602694 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:35:37.603144 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:35:37.603148 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:35:37.603151 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:35:37.603449 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:35:37.604347 (+ 898us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:35:37.604356 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:35:37.604860 (+ 504us) client_negotiation.cc:769] Sending connection context
0926 16:35:37.605183 (+ 323us) client_negotiation.cc:240] Negotiation successful
0926 16:35:37.605706 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":296,"threads_started":1}
I20260926 16:35:37.608217 23892 heartbeater.cc:347] Connected to a master server at 127.6.33.62:43779
I20260926 16:35:37.608537 23892 heartbeater.cc:464] Registering TS with master...
I20260926 16:35:37.609354 23892 heartbeater.cc:511] Master 127.6.33.62:43779 requested a full tablet report, sending...
I20260926 16:35:37.610853 23435 ts_manager.cc:194] Registered new tserver with Master: b7e1b2c89bfb4344b9c72054f28c5ac8 (127.6.33.3:41267)
I20260926 16:35:37.611764 23435 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.33.3@KRBTEST.COM'} at 127.6.33.3:58217
I20260926 16:35:37.617902 6276 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:35:37.637669 23894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:37.622897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57484 (local address 127.6.33.62:43779)
0926 16:35:37.623192 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:37.623200 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:37.623606 (+ 406us) server_negotiation.cc:407] Connection header received
0926 16:35:37.623817 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:37.623822 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:37.623901 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:37.624006 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:35:37.625613 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:37.626349 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:37.628297 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:37.628963 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:37.629098 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:37.632575 (+ 3477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:35:37.632580 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:35:37.632584 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:35:37.632622 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:35:37.635133 (+ 2511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:35:37.635772 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:35:37.635776 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:35:37.635778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:35:37.635891 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:35:37.636305 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:35:37.636308 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:35:37.636310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:35:37.636620 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:35:37.636873 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:37.637223 (+ 350us) server_negotiation.cc:299] Negotiation successful
0926 16:35:37.637412 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 16:35:37.642081 23435 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57484:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:35:37.649435 23435 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:35:37.677794 23905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:37.662939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.2:45723 (local address 127.0.0.1:53460)
0926 16:35:37.664334 (+ 1395us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:37.664397 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 16:35:37.664624 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:35:37.667386 (+ 2762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:35:37.667399 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:35:37.667880 (+ 481us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:35:37.668509 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:37.668539 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:37.671597 (+ 3058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:37.671604 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:37.674403 (+ 2799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:37.674420 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:37.676598 (+ 2178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:37.676605 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:37.676865 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:37.676930 (+ 65us) client_negotiation.cc:769] Sending connection context
0926 16:35:37.677107 (+ 177us) client_negotiation.cc:240] Negotiation successful
0926 16:35:37.677347 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1054,"mutex_wait_us":1,"thread_start_us":870,"threads_started":1}
I20260926 16:35:37.678090 23907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:37.663647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:44979 (local address 127.0.0.1:46726)
0926 16:35:37.669228 (+ 5581us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:37.669252 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 16:35:37.669402 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:35:37.670231 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:35:37.670240 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:35:37.670294 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:35:37.670892 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:37.670904 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:37.673020 (+ 2116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:37.673027 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:37.674800 (+ 1773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:37.674818 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:37.677174 (+ 2356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:37.677181 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:37.677370 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:37.677404 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 16:35:37.677494 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 16:35:37.677646 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5386,"thread_start_us":5224,"threads_started":1}
I20260926 16:35:37.678421 23908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:37.664275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46726 (local address 127.6.33.1:44979)
0926 16:35:37.665552 (+ 1277us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:37.665564 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:37.669547 (+ 3983us) server_negotiation.cc:407] Connection header received
0926 16:35:37.669667 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:37.669675 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:37.669976 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:37.670209 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:35:37.671148 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:37.672898 (+ 1750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:37.675013 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:37.676599 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:37.676722 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:37.677015 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:37.677895 (+ 880us) server_negotiation.cc:299] Negotiation successful
0926 16:35:37.678083 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1033,"thread_start_us":858,"threads_started":1}
I20260926 16:35:37.679284 23906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:37.663297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53460 (local address 127.6.33.2:45723)
0926 16:35:37.666516 (+ 3219us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:37.666539 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:37.666573 (+ 34us) server_negotiation.cc:407] Connection header received
0926 16:35:37.666693 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:37.666705 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:37.667057 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:37.667514 (+ 457us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:35:37.669731 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:37.671422 (+ 1691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:37.674720 (+ 3298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:37.675892 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:37.676062 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:37.676317 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:37.678758 (+ 2441us) server_negotiation.cc:299] Negotiation successful
0926 16:35:37.678957 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3058,"thread_start_us":2950,"threads_started":1}
I20260926 16:35:37.680068 23910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:37.664722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.3:41267 (local address 127.0.0.1:60680)
0926 16:35:37.670898 (+ 6176us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:37.670922 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 16:35:37.671056 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:35:37.672202 (+ 1146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:35:37.672207 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:35:37.672240 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:35:37.672709 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:37.672721 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:37.674907 (+ 2186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:37.674915 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:37.677053 (+ 2138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:37.677071 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:37.679261 (+ 2190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:37.679272 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:37.679474 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:37.679507 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 16:35:37.679592 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 16:35:37.679738 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6017,"thread_start_us":4839,"threads_started":1}
I20260926 16:35:37.680322 23909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:37.665248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60680 (local address 127.6.33.3:41267)
0926 16:35:37.669922 (+ 4674us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:37.669935 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:37.671058 (+ 1123us) server_negotiation.cc:407] Connection header received
0926 16:35:37.671206 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:37.671214 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:37.671546 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:37.672035 (+ 489us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:35:37.673198 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:37.674761 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:37.677280 (+ 2519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:37.678772 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:37.678918 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:37.679127 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:37.679707 (+ 580us) server_negotiation.cc:299] Negotiation successful
0926 16:35:37.679875 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4339,"thread_start_us":4147,"threads_started":1}
I20260926 16:35:37.682898 23693 tablet_service.cc:1467] Processing CreateTablet for tablet a3a46fb9545440c5b329edc027b32734 (DEFAULT_TABLE table=test-table [id=08982c2b96f74e57ab645b0369a6149f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:35:37.682898 23559 tablet_service.cc:1467] Processing CreateTablet for tablet a3a46fb9545440c5b329edc027b32734 (DEFAULT_TABLE table=test-table [id=08982c2b96f74e57ab645b0369a6149f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:35:37.683473 23827 tablet_service.cc:1467] Processing CreateTablet for tablet a3a46fb9545440c5b329edc027b32734 (DEFAULT_TABLE table=test-table [id=08982c2b96f74e57ab645b0369a6149f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:35:37.684463 23559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3a46fb9545440c5b329edc027b32734. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:35:37.684469 23693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3a46fb9545440c5b329edc027b32734. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:35:37.684852 23827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3a46fb9545440c5b329edc027b32734. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:35:37.697659 23911 tablet_bootstrap.cc:492] T a3a46fb9545440c5b329edc027b32734 P 1cdf861ba35547568ba6bda82753dfac: Bootstrap starting.
I20260926 16:35:37.699985 23912 tablet_bootstrap.cc:492] T a3a46fb9545440c5b329edc027b32734 P b7e1b2c89bfb4344b9c72054f28c5ac8: Bootstrap starting.
I20260926 16:35:37.701345 23913 tablet_bootstrap.cc:492] T a3a46fb9545440c5b329edc027b32734 P cd25c2b8cb904630b400168f25ef0db8: Bootstrap starting.
I20260926 16:35:37.702142 23911 tablet_bootstrap.cc:654] T a3a46fb9545440c5b329edc027b32734 P 1cdf861ba35547568ba6bda82753dfac: Neither blocks nor log segments found. Creating new log.
I20260926 16:35:37.704038 23911 log.cc:824] T a3a46fb9545440c5b329edc027b32734 P 1cdf861ba35547568ba6bda82753dfac: Log is configured to *not* fsync() on all Append() calls
I20260926 16:35:37.704572 23912 tablet_bootstrap.cc:654] T a3a46fb9545440c5b329edc027b32734 P b7e1b2c89bfb4344b9c72054f28c5ac8: Neither blocks nor log segments found. Creating new log.
I20260926 16:35:37.704816 23913 tablet_bootstrap.cc:654] T a3a46fb9545440c5b329edc027b32734 P cd25c2b8cb904630b400168f25ef0db8: Neither blocks nor log segments found. Creating new log.
I20260926 16:35:37.705866 23913 log.cc:824] T a3a46fb9545440c5b329edc027b32734 P cd25c2b8cb904630b400168f25ef0db8: Log is configured to *not* fsync() on all Append() calls
I20260926 16:35:37.706208 23912 log.cc:824] T a3a46fb9545440c5b329edc027b32734 P b7e1b2c89bfb4344b9c72054f28c5ac8: Log is configured to *not* fsync() on all Append() calls
I20260926 16:35:37.708748 23911 tablet_bootstrap.cc:492] T a3a46fb9545440c5b329edc027b32734 P 1cdf861ba35547568ba6bda82753dfac: No bootstrap required, opened a new log
I20260926 16:35:37.709071 23913 tablet_bootstrap.cc:492] T a3a46fb9545440c5b329edc027b32734 P cd25c2b8cb904630b400168f25ef0db8: No bootstrap required, opened a new log
I20260926 16:35:37.709111 23911 ts_tablet_manager.cc:1397] T a3a46fb9545440c5b329edc027b32734 P 1cdf861ba35547568ba6bda82753dfac: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260926 16:35:37.709370 23913 ts_tablet_manager.cc:1397] T a3a46fb9545440c5b329edc027b32734 P cd25c2b8cb904630b400168f25ef0db8: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.002s
I20260926 16:35:37.709556 23912 tablet_bootstrap.cc:492] T a3a46fb9545440c5b329edc027b32734 P b7e1b2c89bfb4344b9c72054f28c5ac8: No bootstrap required, opened a new log
I20260926 16:35:37.709827 23912 ts_tablet_manager.cc:1397] T a3a46fb9545440c5b329edc027b32734 P b7e1b2c89bfb4344b9c72054f28c5ac8: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260926 16:35:37.716987 23911 raft_consensus.cc:348] T a3a46fb9545440c5b329edc027b32734 P 1cdf861ba35547568ba6bda82753dfac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cdf861ba35547568ba6bda82753dfac" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 45723 } } peers { permanent_uuid: "b7e1b2c89bfb4344b9c72054f28c5ac8" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 41267 } } peers { permanent_uuid: "cd25c2b8cb904630b400168f25ef0db8" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 44979 } }
I20260926 16:35:37.717458 23911 raft_consensus.cc:374] T a3a46fb9545440c5b329edc027b32734 P 1cdf861ba35547568ba6bda82753dfac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:35:37.717585 23911 raft_consensus.cc:729] T a3a46fb9545440c5b329edc027b32734 P 1cdf861ba35547568ba6bda82753dfac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cdf861ba35547568ba6bda82753dfac, State: Initialized, Role: FOLLOWER
I20260926 16:35:37.718248 23911 consensus_queue.cc:260] T a3a46fb9545440c5b329edc027b32734 P 1cdf861ba35547568ba6bda82753dfac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cdf861ba35547568ba6bda82753dfac" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 45723 } } peers { permanent_uuid: "b7e1b2c89bfb4344b9c72054f28c5ac8" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 41267 } } peers { permanent_uuid: "cd25c2b8cb904630b400168f25ef0db8" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 44979 } }
I20260926 16:35:37.719453 23758 heartbeater.cc:503] Master 127.6.33.62:43779 was elected leader, sending a full tablet report...
I20260926 16:35:37.720191 23911 ts_tablet_manager.cc:1428] T a3a46fb9545440c5b329edc027b32734 P 1cdf861ba35547568ba6bda82753dfac: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260926 16:35:37.720791 23912 raft_consensus.cc:348] T a3a46fb9545440c5b329edc027b32734 P b7e1b2c89bfb4344b9c72054f28c5ac8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cdf861ba35547568ba6bda82753dfac" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 45723 } } peers { permanent_uuid: "b7e1b2c89bfb4344b9c72054f28c5ac8" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 41267 } } peers { permanent_uuid: "cd25c2b8cb904630b400168f25ef0db8" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 44979 } }
I20260926 16:35:37.720780 23913 raft_consensus.cc:348] T a3a46fb9545440c5b329edc027b32734 P cd25c2b8cb904630b400168f25ef0db8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cdf861ba35547568ba6bda82753dfac" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 45723 } } peers { permanent_uuid: "b7e1b2c89bfb4344b9c72054f28c5ac8" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 41267 } } peers { permanent_uuid: "cd25c2b8cb904630b400168f25ef0db8" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 44979 } }
I20260926 16:35:37.721271 23912 raft_consensus.cc:374] T a3a46fb9545440c5b329edc027b32734 P b7e1b2c89bfb4344b9c72054f28c5ac8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:35:37.721273 23913 raft_consensus.cc:374] T a3a46fb9545440c5b329edc027b32734 P cd25c2b8cb904630b400168f25ef0db8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:35:37.721379 23913 raft_consensus.cc:729] T a3a46fb9545440c5b329edc027b32734 P cd25c2b8cb904630b400168f25ef0db8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd25c2b8cb904630b400168f25ef0db8, State: Initialized, Role: FOLLOWER
I20260926 16:35:37.721379 23912 raft_consensus.cc:729] T a3a46fb9545440c5b329edc027b32734 P b7e1b2c89bfb4344b9c72054f28c5ac8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7e1b2c89bfb4344b9c72054f28c5ac8, State: Initialized, Role: FOLLOWER
I20260926 16:35:37.722038 23912 consensus_queue.cc:260] T a3a46fb9545440c5b329edc027b32734 P b7e1b2c89bfb4344b9c72054f28c5ac8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cdf861ba35547568ba6bda82753dfac" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 45723 } } peers { permanent_uuid: "b7e1b2c89bfb4344b9c72054f28c5ac8" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 41267 } } peers { permanent_uuid: "cd25c2b8cb904630b400168f25ef0db8" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 44979 } }
I20260926 16:35:37.722039 23913 consensus_queue.cc:260] T a3a46fb9545440c5b329edc027b32734 P cd25c2b8cb904630b400168f25ef0db8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cdf861ba35547568ba6bda82753dfac" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 45723 } } peers { permanent_uuid: "b7e1b2c89bfb4344b9c72054f28c5ac8" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 41267 } } peers { permanent_uuid: "cd25c2b8cb904630b400168f25ef0db8" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 44979 } }
I20260926 16:35:37.723737 23892 heartbeater.cc:503] Master 127.6.33.62:43779 was elected leader, sending a full tablet report...
I20260926 16:35:37.724005 23912 ts_tablet_manager.cc:1428] T a3a46fb9545440c5b329edc027b32734 P b7e1b2c89bfb4344b9c72054f28c5ac8: Time spent starting tablet: real 0.014s user 0.006s sys 0.008s
I20260926 16:35:37.725095 23913 ts_tablet_manager.cc:1428] T a3a46fb9545440c5b329edc027b32734 P cd25c2b8cb904630b400168f25ef0db8: Time spent starting tablet: real 0.016s user 0.009s sys 0.006s
W20260926 16:35:37.776734 23625 tablet.cc:2369] T a3a46fb9545440c5b329edc027b32734 P cd25c2b8cb904630b400168f25ef0db8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 16:35:37.805486 23759 tablet.cc:2369] T a3a46fb9545440c5b329edc027b32734 P 1cdf861ba35547568ba6bda82753dfac: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 16:35:37.835902 23893 tablet.cc:2369] T a3a46fb9545440c5b329edc027b32734 P b7e1b2c89bfb4344b9c72054f28c5ac8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:35:38.103124 23918 raft_consensus.cc:482] T a3a46fb9545440c5b329edc027b32734 P cd25c2b8cb904630b400168f25ef0db8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:35:38.103509 23918 raft_consensus.cc:504] T a3a46fb9545440c5b329edc027b32734 P cd25c2b8cb904630b400168f25ef0db8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cdf861ba35547568ba6bda82753dfac" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 45723 } } peers { permanent_uuid: "b7e1b2c89bfb4344b9c72054f28c5ac8" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 41267 } } peers { permanent_uuid: "cd25c2b8cb904630b400168f25ef0db8" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 44979 } }
I20260926 16:35:38.105429 23918 leader_election.cc:288] T a3a46fb9545440c5b329edc027b32734 P cd25c2b8cb904630b400168f25ef0db8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cdf861ba35547568ba6bda82753dfac (127.6.33.2:45723), b7e1b2c89bfb4344b9c72054f28c5ac8 (127.6.33.3:41267)
I20260926 16:35:38.115209 23906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:38.105981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:49287 (local address 127.6.33.2:45723)
0926 16:35:38.106217 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:38.106222 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:38.106682 (+ 460us) server_negotiation.cc:407] Connection header received
0926 16:35:38.106959 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:38.106965 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:38.107048 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:38.107169 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:35:38.108217 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:38.109336 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:38.112582 (+ 3246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:38.113863 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:38.113986 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:38.114089 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:38.114794 (+ 705us) server_negotiation.cc:299] Negotiation successful
0926 16:35:38.114929 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 16:35:38.115209 23909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:38.107042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:48121 (local address 127.6.33.3:41267)
0926 16:35:38.107266 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:38.107272 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:38.107504 (+ 232us) server_negotiation.cc:407] Connection header received
0926 16:35:38.107683 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:38.107686 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:38.107741 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:38.107812 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:35:38.108443 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:38.109480 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:38.112582 (+ 3102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:38.113845 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:38.113986 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:38.114091 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:38.114794 (+ 703us) server_negotiation.cc:299] Negotiation successful
0926 16:35:38.114929 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 16:35:38.115437 23921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:38.106743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.3:41267 (local address 127.6.33.1:48121)
0926 16:35:38.107382 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:38.107401 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 16:35:38.107548 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:35:38.107855 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:35:38.107859 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:35:38.107885 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:35:38.108288 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:38.108297 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:38.109630 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:38.109636 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:38.112386 (+ 2750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:38.112404 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:38.114218 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:38.114229 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:38.114526 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:38.114633 (+ 107us) client_negotiation.cc:769] Sending connection context
0926 16:35:38.114743 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 16:35:38.114934 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"mutex_wait_us":312,"thread_start_us":315,"threads_started":1}
I20260926 16:35:38.115460 23920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:38.105810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.2:45723 (local address 127.6.33.1:49287)
0926 16:35:38.106550 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:38.106571 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 16:35:38.106722 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:35:38.107608 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:35:38.107616 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:35:38.107643 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:35:38.108060 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:38.108078 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:38.109487 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:38.109491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:38.112386 (+ 2895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:38.112404 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:38.114254 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:38.114259 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:38.114530 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:38.114656 (+ 126us) client_negotiation.cc:769] Sending connection context
0926 16:35:38.114743 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 16:35:38.114934 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"mutex_wait_us":339,"thread_start_us":411,"threads_started":1}
I20260926 16:35:38.116612 23847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3a46fb9545440c5b329edc027b32734" candidate_uuid: "cd25c2b8cb904630b400168f25ef0db8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7e1b2c89bfb4344b9c72054f28c5ac8" is_pre_election: true
I20260926 16:35:38.117071 23713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3a46fb9545440c5b329edc027b32734" candidate_uuid: "cd25c2b8cb904630b400168f25ef0db8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cdf861ba35547568ba6bda82753dfac" is_pre_election: true
I20260926 16:35:38.117179 23847 raft_consensus.cc:2454] T a3a46fb9545440c5b329edc027b32734 P b7e1b2c89bfb4344b9c72054f28c5ac8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd25c2b8cb904630b400168f25ef0db8 in term 0.
I20260926 16:35:38.117571 23713 raft_consensus.cc:2454] T a3a46fb9545440c5b329edc027b32734 P 1cdf861ba35547568ba6bda82753dfac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd25c2b8cb904630b400168f25ef0db8 in term 0.
I20260926 16:35:38.117928 23514 leader_election.cc:302] T a3a46fb9545440c5b329edc027b32734 P cd25c2b8cb904630b400168f25ef0db8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7e1b2c89bfb4344b9c72054f28c5ac8, cd25c2b8cb904630b400168f25ef0db8; no voters:
I20260926 16:35:38.118356 23918 raft_consensus.cc:2781] T a3a46fb9545440c5b329edc027b32734 P cd25c2b8cb904630b400168f25ef0db8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:35:38.118537 23918 raft_consensus.cc:482] T a3a46fb9545440c5b329edc027b32734 P cd25c2b8cb904630b400168f25ef0db8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:35:38.118666 23918 raft_consensus.cc:3037] T a3a46fb9545440c5b329edc027b32734 P cd25c2b8cb904630b400168f25ef0db8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:35:38.121058 23918 raft_consensus.cc:504] T a3a46fb9545440c5b329edc027b32734 P cd25c2b8cb904630b400168f25ef0db8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cdf861ba35547568ba6bda82753dfac" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 45723 } } peers { permanent_uuid: "b7e1b2c89bfb4344b9c72054f28c5ac8" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 41267 } } peers { permanent_uuid: "cd25c2b8cb904630b400168f25ef0db8" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 44979 } }
I20260926 16:35:38.121672 23918 leader_election.cc:288] T a3a46fb9545440c5b329edc027b32734 P cd25c2b8cb904630b400168f25ef0db8 [CANDIDATE]: Term 1 election: Requested vote from peers 1cdf861ba35547568ba6bda82753dfac (127.6.33.2:45723), b7e1b2c89bfb4344b9c72054f28c5ac8 (127.6.33.3:41267)
I20260926 16:35:38.122318 23713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3a46fb9545440c5b329edc027b32734" candidate_uuid: "cd25c2b8cb904630b400168f25ef0db8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cdf861ba35547568ba6bda82753dfac"
I20260926 16:35:38.122321 23847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3a46fb9545440c5b329edc027b32734" candidate_uuid: "cd25c2b8cb904630b400168f25ef0db8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7e1b2c89bfb4344b9c72054f28c5ac8"
I20260926 16:35:38.122536 23847 raft_consensus.cc:3037] T a3a46fb9545440c5b329edc027b32734 P b7e1b2c89bfb4344b9c72054f28c5ac8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:35:38.122537 23713 raft_consensus.cc:3037] T a3a46fb9545440c5b329edc027b32734 P 1cdf861ba35547568ba6bda82753dfac [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:35:38.125802 23847 raft_consensus.cc:2454] T a3a46fb9545440c5b329edc027b32734 P b7e1b2c89bfb4344b9c72054f28c5ac8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd25c2b8cb904630b400168f25ef0db8 in term 1.
I20260926 16:35:38.125802 23713 raft_consensus.cc:2454] T a3a46fb9545440c5b329edc027b32734 P 1cdf861ba35547568ba6bda82753dfac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd25c2b8cb904630b400168f25ef0db8 in term 1.
I20260926 16:35:38.126204 23514 leader_election.cc:302] T a3a46fb9545440c5b329edc027b32734 P cd25c2b8cb904630b400168f25ef0db8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cdf861ba35547568ba6bda82753dfac, cd25c2b8cb904630b400168f25ef0db8; no voters:
I20260926 16:35:38.126475 23918 raft_consensus.cc:2781] T a3a46fb9545440c5b329edc027b32734 P cd25c2b8cb904630b400168f25ef0db8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:35:38.126915 23918 raft_consensus.cc:686] T a3a46fb9545440c5b329edc027b32734 P cd25c2b8cb904630b400168f25ef0db8 [term 1 LEADER]: Becoming Leader. State: Replica: cd25c2b8cb904630b400168f25ef0db8, State: Running, Role: LEADER
I20260926 16:35:38.127504 23918 consensus_queue.cc:237] T a3a46fb9545440c5b329edc027b32734 P cd25c2b8cb904630b400168f25ef0db8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cdf861ba35547568ba6bda82753dfac" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 45723 } } peers { permanent_uuid: "b7e1b2c89bfb4344b9c72054f28c5ac8" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 41267 } } peers { permanent_uuid: "cd25c2b8cb904630b400168f25ef0db8" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 44979 } }
I20260926 16:35:38.132378 23434 catalog_manager.cc:5495] T a3a46fb9545440c5b329edc027b32734 P cd25c2b8cb904630b400168f25ef0db8 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd25c2b8cb904630b400168f25ef0db8 (127.6.33.1). New cstate: current_term: 1 leader_uuid: "cd25c2b8cb904630b400168f25ef0db8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cdf861ba35547568ba6bda82753dfac" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 45723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7e1b2c89bfb4344b9c72054f28c5ac8" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd25c2b8cb904630b400168f25ef0db8" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 44979 } health_report { overall_health: HEALTHY } } }
I20260926 16:35:38.253815 23926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:38.247901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46732 (local address 127.6.33.1:44979)
0926 16:35:38.248407 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:38.248412 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:38.248560 (+ 148us) server_negotiation.cc:407] Connection header received
0926 16:35:38.248737 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:38.248741 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:38.248878 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:38.249021 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:35:38.249447 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:38.250149 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:38.251783 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:38.252193 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:38.252287 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:38.252731 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:35:38.253347 (+ 616us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:35:38.253463 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:38.253527 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 16:35:38.253593 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":304,"threads_started":1}
I20260926 16:35:38.262115 23847 raft_consensus.cc:1261] T a3a46fb9545440c5b329edc027b32734 P b7e1b2c89bfb4344b9c72054f28c5ac8 [term 1 FOLLOWER]: Refusing update from remote peer cd25c2b8cb904630b400168f25ef0db8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:35:38.262135 23713 raft_consensus.cc:1261] T a3a46fb9545440c5b329edc027b32734 P 1cdf861ba35547568ba6bda82753dfac [term 1 FOLLOWER]: Refusing update from remote peer cd25c2b8cb904630b400168f25ef0db8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:35:38.263139 23918 consensus_queue.cc:1035] T a3a46fb9545440c5b329edc027b32734 P cd25c2b8cb904630b400168f25ef0db8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cdf861ba35547568ba6bda82753dfac" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 45723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:35:38.263541 23922 consensus_queue.cc:1035] T a3a46fb9545440c5b329edc027b32734 P cd25c2b8cb904630b400168f25ef0db8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7e1b2c89bfb4344b9c72054f28c5ac8" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:35:38.290447 23433 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57484:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:35:38.290735 23433 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57484:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:35:38.294421 23433 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 71ff3629338140bfb9414b7ff046ec9a: Sending DeleteTablet for 3 replicas of tablet a3a46fb9545440c5b329edc027b32734
I20260926 16:35:38.295687 23693 tablet_service.cc:1514] Processing DeleteTablet for tablet a3a46fb9545440c5b329edc027b32734 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:35:38 UTC) from {username='kudu', principal='kudu/127.6.33.62@KRBTEST.COM'} at 127.0.0.1:53460
I20260926 16:35:38.295781 23827 tablet_service.cc:1514] Processing DeleteTablet for tablet a3a46fb9545440c5b329edc027b32734 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:35:38 UTC) from {username='kudu', principal='kudu/127.6.33.62@KRBTEST.COM'} at 127.0.0.1:60680
I20260926 16:35:38.295900 23559 tablet_service.cc:1514] Processing DeleteTablet for tablet a3a46fb9545440c5b329edc027b32734 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:35:38 UTC) from {username='kudu', principal='kudu/127.6.33.62@KRBTEST.COM'} at 127.0.0.1:46726
I20260926 16:35:38.297380 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 23496
Sep 26 16:35:38 dist-test-slave-khd3 krb5kdc[23390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.1@KRBTEST.COM
I20260926 16:35:38.316273 23940 tablet_replica.cc:330] T a3a46fb9545440c5b329edc027b32734 P b7e1b2c89bfb4344b9c72054f28c5ac8: stopping tablet replica
I20260926 16:35:38.317147 23939 tablet_replica.cc:330] T a3a46fb9545440c5b329edc027b32734 P 1cdf861ba35547568ba6bda82753dfac: stopping tablet replica
I20260926 16:35:38.317760 23939 raft_consensus.cc:2229] T a3a46fb9545440c5b329edc027b32734 P 1cdf861ba35547568ba6bda82753dfac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:35:38.318311 23940 raft_consensus.cc:2229] T a3a46fb9545440c5b329edc027b32734 P b7e1b2c89bfb4344b9c72054f28c5ac8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:35:38.318926 23939 raft_consensus.cc:2258] T a3a46fb9545440c5b329edc027b32734 P 1cdf861ba35547568ba6bda82753dfac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:35:38.319438 23940 raft_consensus.cc:2258] T a3a46fb9545440c5b329edc027b32734 P b7e1b2c89bfb4344b9c72054f28c5ac8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:35:38.321831 23940 ts_tablet_manager.cc:1905] T a3a46fb9545440c5b329edc027b32734 P b7e1b2c89bfb4344b9c72054f28c5ac8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:35:38.323565 23939 ts_tablet_manager.cc:1905] T a3a46fb9545440c5b329edc027b32734 P 1cdf861ba35547568ba6bda82753dfac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:35:38.327071 23926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:38.298923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46746 (local address 127.6.33.1:44979)
0926 16:35:38.299105 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:38.299111 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:38.299990 (+ 879us) server_negotiation.cc:407] Connection header received
0926 16:35:38.300329 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:38.300338 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:38.300445 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:38.300535 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:35:38.303014 (+ 2479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:38.304804 (+ 1790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:38.307570 (+ 2766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:38.308555 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:38.308688 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:38.316330 (+ 7642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:35:38.316355 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:35:38.316383 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:35:38.316450 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:35:38.322128 (+ 5678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:35:38.323402 (+ 1274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:35:38.323416 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:35:38.323422 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:35:38.323634 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:35:38.324225 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:35:38.324233 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:35:38.324242 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:35:38.324855 (+ 613us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:35:38.325247 (+ 392us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:38.326057 (+ 810us) server_negotiation.cc:299] Negotiation successful
0926 16:35:38.326501 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 16:35:38.328842 23619 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:35:38.328881 23941 tablet_replica.cc:330] T a3a46fb9545440c5b329edc027b32734 P cd25c2b8cb904630b400168f25ef0db8: stopping tablet replica
I20260926 16:35:38.334800 23939 ts_tablet_manager.cc:1918] T a3a46fb9545440c5b329edc027b32734 P 1cdf861ba35547568ba6bda82753dfac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:35:38.334838 23940 ts_tablet_manager.cc:1918] T a3a46fb9545440c5b329edc027b32734 P b7e1b2c89bfb4344b9c72054f28c5ac8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:35:38.334986 23939 log.cc:1196] T a3a46fb9545440c5b329edc027b32734 P 1cdf861ba35547568ba6bda82753dfac: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/ts-1/wal/wals/a3a46fb9545440c5b329edc027b32734
I20260926 16:35:38.334997 23940 log.cc:1196] T a3a46fb9545440c5b329edc027b32734 P b7e1b2c89bfb4344b9c72054f28c5ac8: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440233850748-6276-0/minicluster-data/ts-2/wal/wals/a3a46fb9545440c5b329edc027b32734
I20260926 16:35:38.335420 23939 ts_tablet_manager.cc:1939] T a3a46fb9545440c5b329edc027b32734 P 1cdf861ba35547568ba6bda82753dfac: Deleting consensus metadata
I20260926 16:35:38.335515 23940 ts_tablet_manager.cc:1939] T a3a46fb9545440c5b329edc027b32734 P b7e1b2c89bfb4344b9c72054f28c5ac8: Deleting consensus metadata
I20260926 16:35:38.337184 23421 catalog_manager.cc:4841] TS 1cdf861ba35547568ba6bda82753dfac (127.6.33.2:45723): tablet a3a46fb9545440c5b329edc027b32734 (table test-table [id=08982c2b96f74e57ab645b0369a6149f]) successfully deleted
I20260926 16:35:38.337535 23421 catalog_manager.cc:4841] TS b7e1b2c89bfb4344b9c72054f28c5ac8 (127.6.33.3:41267): tablet a3a46fb9545440c5b329edc027b32734 (table test-table [id=08982c2b96f74e57ab645b0369a6149f]) successfully deleted
I20260926 16:35:38.329384 23941 raft_consensus.cc:2229] T a3a46fb9545440c5b329edc027b32734 P cd25c2b8cb904630b400168f25ef0db8 [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:35:38.427448 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 23496
W20260926 16:35:38.444922 23419 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.33.1:44979 (error 108)
I20260926 16:35:38.445747 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 23630
I20260926 16:35:38.447103 23945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:38.446293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:44979 (local address 127.0.0.1:46756)
0926 16:35:38.446731 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:38.446883 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.1:44979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":511,"threads_started":1}
W20260926 16:35:38.447424 23419 catalog_manager.cc:4584] TS cd25c2b8cb904630b400168f25ef0db8 (127.6.33.1:44979): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a3a46fb9545440c5b329edc027b32734: Network error: Client connection negotiation failed: client connection to 127.6.33.1:44979: connect: Connection refused (error 111)
Sep 26 16:35:38 dist-test-slave-khd3 krb5kdc[23390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.2@KRBTEST.COM
I20260926 16:35:38.462309 23906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:38.446570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53476 (local address 127.6.33.2:45723)
0926 16:35:38.446760 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:38.446767 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:38.446789 (+ 22us) server_negotiation.cc:407] Connection header received
0926 16:35:38.446962 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:38.446967 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:38.447041 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:38.447148 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:35:38.448462 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:38.449349 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:38.450744 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:38.451202 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:38.451336 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:38.455547 (+ 4211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:35:38.455562 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:35:38.455577 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:35:38.455612 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:35:38.459197 (+ 3585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:35:38.460011 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:35:38.460019 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:35:38.460021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:35:38.460163 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:35:38.460586 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:35:38.460590 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:35:38.460592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:35:38.461044 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:35:38.461396 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:38.461711 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 16:35:38.461950 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 16:35:38.463124 23753 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:35:38.501295 23945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:38.500713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:44979 (local address 127.0.0.1:46772)
0926 16:35:38.500992 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:38.501114 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.1:44979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260926 16:35:38.502419 23945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:38.501987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:44979 (local address 127.0.0.1:46784)
0926 16:35:38.502163 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:38.502249 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.1:44979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 16:35:38.554150 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 23630
I20260926 16:35:38.562938 23945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:38.562464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:44979 (local address 127.0.0.1:46786)
0926 16:35:38.562686 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:38.562793 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.1:44979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260926 16:35:38.564030 23945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:38.563671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:44979 (local address 127.0.0.1:46788)
0926 16:35:38.563820 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:38.563898 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.1:44979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 16:35:38.571890 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 23764
Sep 26 16:35:38 dist-test-slave-khd3 krb5kdc[23390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.3@KRBTEST.COM
I20260926 16:35:38.589653 23909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:38.572724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60690 (local address 127.6.33.3:41267)
0926 16:35:38.572922 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:38.572927 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:38.572946 (+ 19us) server_negotiation.cc:407] Connection header received
0926 16:35:38.573187 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:38.573192 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:38.573309 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:38.573404 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:35:38.574868 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:38.575679 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:38.577270 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:38.577678 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:38.577785 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:38.582611 (+ 4826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:35:38.582630 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:35:38.582645 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:35:38.582690 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:35:38.586400 (+ 3710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:35:38.587370 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:35:38.587377 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:35:38.587380 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:35:38.587551 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:35:38.587981 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:35:38.587986 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:35:38.587988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:35:38.588360 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:35:38.588668 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:38.589076 (+ 408us) server_negotiation.cc:299] Negotiation successful
0926 16:35:38.589304 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 16:35:38.590363 23887 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:35:38.644675 23945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:38.644136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:44979 (local address 127.0.0.1:46804)
0926 16:35:38.644395 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:38.644495 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.1:44979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260926 16:35:38.646023 23945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:38.645561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:44979 (local address 127.0.0.1:46818)
0926 16:35:38.645769 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:38.645839 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.1:44979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260926 16:35:38.684046 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 23764
I20260926 16:35:38.700621 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 23403
I20260926 16:35:38.701926 23465 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:35:38.814373 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 23403
2026-09-26T16:35:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4778 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-26T16:35:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:35:38Z Disabled control of system clock
I20260926 16:35:38.897194 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:40795
--webserver_interface=127.6.33.62
--webserver_port=0
--builtin_ntp_servers=127.6.33.20:35139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.33.62:40795
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 16:35:39.238175 23952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:35:39.238505 23952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:35:39.238590 23952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:35:39.246008 23952 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:35:39.246133 23952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:35:39.246171 23952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:35:39.246227 23952 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:35:39.246268 23952 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:35:39.255844 23952 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:35139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.33.62:40795
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:40795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:35:39.257232 23952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:35:39.259002 23952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:35:39.271739 23960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:35:39.271801 23957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:35:39.271777 23958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:35:39.272681 23952 server_base.cc:1034] running on GCE node
I20260926 16:35:39.273607 23952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:35:39.275098 23952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:35:39.276314 23952 hybrid_clock.cc:648] HybridClock initialized: now 1790440539276267 us; error 35 us; skew 500 ppm
I20260926 16:35:39.276712 23952 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:35:39.279336 23952 webserver.cc:460] Webserver started at http://127.6.33.62:37265/ using document root <none> and password file <none>
I20260926 16:35:39.280273 23952 fs_manager.cc:359] Metadata directory not provided
I20260926 16:35:39.280375 23952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:35:39.280797 23952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:35:39.283918 23952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/master-0/data/instance:
uuid: "81fef6a4e57045b8802a7f9b0c327ee6"
format_stamp: "Formatted at 2026-09-26 16:35:39 on dist-test-slave-khd3"
server_key: "50b821b40c96bd955b801fd4dd52afc5"
server_key_iv: "f4a54616fe1f5cfa81abbf8d88533ce5"
server_key_version: "encryptionkey@0"
I20260926 16:35:39.284861 23952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/master-0/wal/instance:
uuid: "81fef6a4e57045b8802a7f9b0c327ee6"
format_stamp: "Formatted at 2026-09-26 16:35:39 on dist-test-slave-khd3"
server_key: "50b821b40c96bd955b801fd4dd52afc5"
server_key_iv: "f4a54616fe1f5cfa81abbf8d88533ce5"
server_key_version: "encryptionkey@0"
I20260926 16:35:39.290639 23952 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 16:35:39.295010 23966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:35:39.296868 23952 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 16:35:39.297154 23952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/master-0/wal
uuid: "81fef6a4e57045b8802a7f9b0c327ee6"
format_stamp: "Formatted at 2026-09-26 16:35:39 on dist-test-slave-khd3"
server_key: "50b821b40c96bd955b801fd4dd52afc5"
server_key_iv: "f4a54616fe1f5cfa81abbf8d88533ce5"
server_key_version: "encryptionkey@0"
I20260926 16:35:39.297418 23952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:35:39.321909 23952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:35:39.322888 23952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:35:39.323210 23952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:35:39.343216 23952 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.62:40795
I20260926 16:35:39.343235 24017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.62:40795 every 8 connection(s)
I20260926 16:35:39.344662 23952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
I20260926 16:35:39.348618 24018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:35:39.351711 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 23952
I20260926 16:35:39.351897 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/master-0/wal/instance
I20260926 16:35:39.352470 6276 external_mini_cluster.cc:1428] Setting key 7a920b9e26bc97bf71aa35fef77885ef
I20260926 16:35:39.361855 24018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81fef6a4e57045b8802a7f9b0c327ee6: Bootstrap starting.
I20260926 16:35:39.365080 24018 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81fef6a4e57045b8802a7f9b0c327ee6: Neither blocks nor log segments found. Creating new log.
I20260926 16:35:39.366276 24018 log.cc:824] T 00000000000000000000000000000000 P 81fef6a4e57045b8802a7f9b0c327ee6: Log is configured to *not* fsync() on all Append() calls
I20260926 16:35:39.366623 24021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:39.354707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43588 (local address 127.6.33.62:40795)
0926 16:35:39.355820 (+ 1113us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:39.355832 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:39.355875 (+ 43us) server_negotiation.cc:407] Connection header received
0926 16:35:39.357670 (+ 1795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:39.357694 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:39.358178 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:39.358623 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:35:39.359248 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:39.360619 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:39.362125 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:39.362824 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:39.363021 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:39.364235 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:35:39.364242 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:35:39.364255 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:35:39.364850 (+ 595us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:35:39.364944 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:35:39.364964 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:35:39.365099 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:39.365319 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 16:35:39.365632 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":440,"threads_started":1}
I20260926 16:35:39.369311 24018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81fef6a4e57045b8802a7f9b0c327ee6: No bootstrap required, opened a new log
I20260926 16:35:39.377214 24018 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81fef6a4e57045b8802a7f9b0c327ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81fef6a4e57045b8802a7f9b0c327ee6" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 40795 } }
I20260926 16:35:39.377617 24018 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81fef6a4e57045b8802a7f9b0c327ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:35:39.377697 24018 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81fef6a4e57045b8802a7f9b0c327ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81fef6a4e57045b8802a7f9b0c327ee6, State: Initialized, Role: FOLLOWER
I20260926 16:35:39.378337 24018 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81fef6a4e57045b8802a7f9b0c327ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81fef6a4e57045b8802a7f9b0c327ee6" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 40795 } }
I20260926 16:35:39.378568 24018 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81fef6a4e57045b8802a7f9b0c327ee6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:35:39.378717 24018 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81fef6a4e57045b8802a7f9b0c327ee6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:35:39.378860 24018 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81fef6a4e57045b8802a7f9b0c327ee6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:35:39.381699 24018 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81fef6a4e57045b8802a7f9b0c327ee6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81fef6a4e57045b8802a7f9b0c327ee6" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 40795 } }
I20260926 16:35:39.382232 24018 leader_election.cc:302] T 00000000000000000000000000000000 P 81fef6a4e57045b8802a7f9b0c327ee6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81fef6a4e57045b8802a7f9b0c327ee6; no voters:
I20260926 16:35:39.382848 24018 leader_election.cc:288] T 00000000000000000000000000000000 P 81fef6a4e57045b8802a7f9b0c327ee6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:35:39.383046 24023 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81fef6a4e57045b8802a7f9b0c327ee6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:35:39.383809 24023 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81fef6a4e57045b8802a7f9b0c327ee6 [term 1 LEADER]: Becoming Leader. State: Replica: 81fef6a4e57045b8802a7f9b0c327ee6, State: Running, Role: LEADER
I20260926 16:35:39.384791 24023 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81fef6a4e57045b8802a7f9b0c327ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81fef6a4e57045b8802a7f9b0c327ee6" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 40795 } }
I20260926 16:35:39.385282 24018 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81fef6a4e57045b8802a7f9b0c327ee6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:35:39.387998 24025 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81fef6a4e57045b8802a7f9b0c327ee6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81fef6a4e57045b8802a7f9b0c327ee6. Latest consensus state: current_term: 1 leader_uuid: "81fef6a4e57045b8802a7f9b0c327ee6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81fef6a4e57045b8802a7f9b0c327ee6" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 40795 } } }
I20260926 16:35:39.388473 24025 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81fef6a4e57045b8802a7f9b0c327ee6 [sys.catalog]: This master's current role is: LEADER
I20260926 16:35:39.388967 24024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81fef6a4e57045b8802a7f9b0c327ee6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81fef6a4e57045b8802a7f9b0c327ee6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81fef6a4e57045b8802a7f9b0c327ee6" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 40795 } } }
I20260926 16:35:39.389394 24024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81fef6a4e57045b8802a7f9b0c327ee6 [sys.catalog]: This master's current role is: LEADER
I20260926 16:35:39.391198 24030 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:35:39.399035 24030 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:35:39.409893 24030 catalog_manager.cc:1347] Generated new cluster ID: 7765f2b9b5c4487eb4ae4d3e3ef32f51
I20260926 16:35:39.410036 24030 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:35:39.423035 24030 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:35:39.424968 24030 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:35:39.436142 24030 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81fef6a4e57045b8802a7f9b0c327ee6: Generated new TSK 0
I20260926 16:35:39.437319 24030 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:35:39.454221 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.1:0
--local_ip_for_outbound_sockets=127.6.33.1
--webserver_interface=127.6.33.1
--webserver_port=0
--tserver_master_addrs=127.6.33.62:40795
--builtin_ntp_servers=127.6.33.20:35139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 16:35:39.807412 24042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:35:39.807744 24042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:35:39.807888 24042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:35:39.815603 24042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:35:39.815733 24042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:35:39.815914 24042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.1
I20260926 16:35:39.825659 24042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:35139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.33.1
--webserver_port=0
--tserver_master_addrs=127.6.33.62:40795
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.1
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:35:39.827194 24042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:35:39.829084 24042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:35:39.843291 24047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:35:39.843246 24050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:35:39.843315 24048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:35:39.844126 24042 server_base.cc:1034] running on GCE node
I20260926 16:35:39.844880 24042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:35:39.845928 24042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:35:39.847170 24042 hybrid_clock.cc:648] HybridClock initialized: now 1790440539847115 us; error 56 us; skew 500 ppm
I20260926 16:35:39.847513 24042 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:35:39.849978 24042 webserver.cc:460] Webserver started at http://127.6.33.1:39699/ using document root <none> and password file <none>
I20260926 16:35:39.850719 24042 fs_manager.cc:359] Metadata directory not provided
I20260926 16:35:39.850853 24042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:35:39.851182 24042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:35:39.854190 24042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/ts-0/data/instance:
uuid: "75c4425b3c0c410795f1ed466357d419"
format_stamp: "Formatted at 2026-09-26 16:35:39 on dist-test-slave-khd3"
server_key: "ff261b9752a5920b822fda667b32256b"
server_key_iv: "ec829a0486710dbc99497dfa47df4453"
server_key_version: "encryptionkey@0"
I20260926 16:35:39.855016 24042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/ts-0/wal/instance:
uuid: "75c4425b3c0c410795f1ed466357d419"
format_stamp: "Formatted at 2026-09-26 16:35:39 on dist-test-slave-khd3"
server_key: "ff261b9752a5920b822fda667b32256b"
server_key_iv: "ec829a0486710dbc99497dfa47df4453"
server_key_version: "encryptionkey@0"
I20260926 16:35:39.860407 24042 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260926 16:35:39.864316 24056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:35:39.865883 24042 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:35:39.866096 24042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/ts-0/wal
uuid: "75c4425b3c0c410795f1ed466357d419"
format_stamp: "Formatted at 2026-09-26 16:35:39 on dist-test-slave-khd3"
server_key: "ff261b9752a5920b822fda667b32256b"
server_key_iv: "ec829a0486710dbc99497dfa47df4453"
server_key_version: "encryptionkey@0"
I20260926 16:35:39.866336 24042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:35:39.880707 24042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:35:39.881680 24042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:35:39.881980 24042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:35:39.883001 24042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:35:39.884600 24042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:35:39.884732 24042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:35:39.884960 24042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:35:39.885037 24042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:35:39.920185 24042 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.1:38861
I20260926 16:35:39.920303 24168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.1:38861 every 8 connection(s)
I20260926 16:35:39.921689 24042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
I20260926 16:35:39.930655 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 24042
I20260926 16:35:39.930866 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/ts-0/wal/instance
I20260926 16:35:39.931350 6276 external_mini_cluster.cc:1428] Setting key d50c31bd788fb821a805f04c51180f41
I20260926 16:35:39.934648 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.2:0
--local_ip_for_outbound_sockets=127.6.33.2
--webserver_interface=127.6.33.2
--webserver_port=0
--tserver_master_addrs=127.6.33.62:40795
--builtin_ntp_servers=127.6.33.20:35139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 16:35:39.936204 24171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:39.924403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:34597 (local address 127.6.33.62:40795)
0926 16:35:39.924911 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:39.924916 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:39.926103 (+ 1187us) server_negotiation.cc:407] Connection header received
0926 16:35:39.928100 (+ 1997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:39.928106 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:39.928259 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:39.928388 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:35:39.929661 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:39.930563 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:39.932571 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:39.933296 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:39.933429 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:39.935280 (+ 1851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:35:39.935287 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:35:39.935294 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:35:39.935324 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:35:39.935406 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:35:39.935425 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:35:39.935527 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:39.935825 (+ 298us) server_negotiation.cc:299] Negotiation successful
0926 16:35:39.935912 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":258,"threads_started":1}
I20260926 16:35:39.937902 24172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:39.924954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:40795 (local address 127.6.33.1:34597)
0926 16:35:39.925916 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:39.925956 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 16:35:39.927768 (+ 1812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:35:39.928681 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:35:39.928691 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:35:39.928854 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:35:39.929478 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:39.929495 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:39.930792 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:39.930800 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:39.932421 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:39.932434 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:39.933591 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:39.933598 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:39.933929 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:39.934832 (+ 903us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:35:39.934837 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:35:39.934902 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:35:39.935016 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:35:39.935560 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:35:39.935576 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:35:39.935588 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 16:35:39.935811 (+ 223us) client_negotiation.cc:240] Negotiation successful
0926 16:35:39.936304 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":293,"threads_started":1}
I20260926 16:35:39.939553 24169 heartbeater.cc:347] Connected to a master server at 127.6.33.62:40795
I20260926 16:35:39.939884 24169 heartbeater.cc:464] Registering TS with master...
I20260926 16:35:39.940723 24169 heartbeater.cc:511] Master 127.6.33.62:40795 requested a full tablet report, sending...
I20260926 16:35:39.942943 23983 ts_manager.cc:194] Registered new tserver with Master: 75c4425b3c0c410795f1ed466357d419 (127.6.33.1:38861)
I20260926 16:35:39.945214 23983 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.33.1:34597
W20260926 16:35:40.283403 24173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:35:40.283725 24173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:35:40.283815 24173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:35:40.291754 24173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:35:40.291868 24173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:35:40.291976 24173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.2
I20260926 16:35:40.301759 24173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:35139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.33.2
--webserver_port=0
--tserver_master_addrs=127.6.33.62:40795
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.2
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:35:40.303234 24173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:35:40.305178 24173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:35:40.319690 24179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:35:40.319615 24181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:35:40.319664 24178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:35:40.320492 24173 server_base.cc:1034] running on GCE node
I20260926 16:35:40.321231 24173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:35:40.322347 24173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:35:40.323596 24173 hybrid_clock.cc:648] HybridClock initialized: now 1790440540323532 us; error 60 us; skew 500 ppm
I20260926 16:35:40.323942 24173 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:35:40.326371 24173 webserver.cc:460] Webserver started at http://127.6.33.2:34481/ using document root <none> and password file <none>
I20260926 16:35:40.327136 24173 fs_manager.cc:359] Metadata directory not provided
I20260926 16:35:40.327275 24173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:35:40.327621 24173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:35:40.330654 24173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/ts-1/data/instance:
uuid: "cde1a2073b514fc8bebc9cc9643ca37d"
format_stamp: "Formatted at 2026-09-26 16:35:40 on dist-test-slave-khd3"
server_key: "60261b183a8a03c3b60081f34e081c1b"
server_key_iv: "cb83c5fc6be2a65145608e15e60e9d06"
server_key_version: "encryptionkey@0"
I20260926 16:35:40.331470 24173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/ts-1/wal/instance:
uuid: "cde1a2073b514fc8bebc9cc9643ca37d"
format_stamp: "Formatted at 2026-09-26 16:35:40 on dist-test-slave-khd3"
server_key: "60261b183a8a03c3b60081f34e081c1b"
server_key_iv: "cb83c5fc6be2a65145608e15e60e9d06"
server_key_version: "encryptionkey@0"
I20260926 16:35:40.336932 24173 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 16:35:40.340948 24187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:35:40.342737 24173 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 16:35:40.342933 24173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/ts-1/wal
uuid: "cde1a2073b514fc8bebc9cc9643ca37d"
format_stamp: "Formatted at 2026-09-26 16:35:40 on dist-test-slave-khd3"
server_key: "60261b183a8a03c3b60081f34e081c1b"
server_key_iv: "cb83c5fc6be2a65145608e15e60e9d06"
server_key_version: "encryptionkey@0"
I20260926 16:35:40.343179 24173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:35:40.358779 24173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:35:40.359683 24173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:35:40.359973 24173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:35:40.361193 24173 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:35:40.362809 24173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:35:40.362895 24173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:35:40.363026 24173 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:35:40.363124 24173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:35:40.397686 24173 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.2:35419
I20260926 16:35:40.397778 24299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.2:35419 every 8 connection(s)
I20260926 16:35:40.399005 24173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
I20260926 16:35:40.403321 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 24173
I20260926 16:35:40.403496 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/ts-1/wal/instance
I20260926 16:35:40.404084 6276 external_mini_cluster.cc:1428] Setting key 4a0c313210a029e99c2aabd964223631
I20260926 16:35:40.407399 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.3:0
--local_ip_for_outbound_sockets=127.6.33.3
--webserver_interface=127.6.33.3
--webserver_port=0
--tserver_master_addrs=127.6.33.62:40795
--builtin_ntp_servers=127.6.33.20:35139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 16:35:40.414448 24171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:40.401773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:40427 (local address 127.6.33.62:40795)
0926 16:35:40.401991 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:40.401998 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:40.403869 (+ 1871us) server_negotiation.cc:407] Connection header received
0926 16:35:40.406220 (+ 2351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:40.406228 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:40.406312 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:40.406411 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:35:40.407975 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:40.408711 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:40.411362 (+ 2651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:40.411759 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:40.411870 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:40.413668 (+ 1798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:35:40.413674 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:35:40.413677 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:35:40.413707 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:35:40.413742 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:35:40.413755 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:35:40.413834 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:40.414118 (+ 284us) server_negotiation.cc:299] Negotiation successful
0926 16:35:40.414209 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 16:35:40.416097 24302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:40.402429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:40795 (local address 127.6.33.2:40427)
0926 16:35:40.403672 (+ 1243us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:40.403728 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 16:35:40.405881 (+ 2153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:35:40.406716 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:35:40.406729 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:35:40.406861 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:35:40.407763 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:40.407792 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:40.408902 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:40.408910 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:40.411146 (+ 2236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:40.411178 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:40.412007 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:40.412016 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:40.412384 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:40.413257 (+ 873us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:35:40.413263 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:35:40.413320 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:35:40.413434 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:35:40.413860 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:35:40.413874 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:35:40.413885 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 16:35:40.414086 (+ 201us) client_negotiation.cc:240] Negotiation successful
0926 16:35:40.414569 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":300,"threads_started":1}
I20260926 16:35:40.417900 24300 heartbeater.cc:347] Connected to a master server at 127.6.33.62:40795
I20260926 16:35:40.418228 24300 heartbeater.cc:464] Registering TS with master...
I20260926 16:35:40.419039 24300 heartbeater.cc:511] Master 127.6.33.62:40795 requested a full tablet report, sending...
I20260926 16:35:40.420610 23983 ts_manager.cc:194] Registered new tserver with Master: cde1a2073b514fc8bebc9cc9643ca37d (127.6.33.2:35419)
I20260926 16:35:40.421486 23983 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.33.2:40427
W20260926 16:35:40.750814 24303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:35:40.751161 24303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:35:40.751331 24303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:35:40.758736 24303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:35:40.758896 24303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:35:40.759088 24303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.3
I20260926 16:35:40.768486 24303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:35139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.33.3
--webserver_port=0
--tserver_master_addrs=127.6.33.62:40795
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.3
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:35:40.769937 24303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:35:40.771916 24303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:35:40.786160 24309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:35:40.786201 24311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:35:40.786116 24308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:35:40.787089 24303 server_base.cc:1034] running on GCE node
I20260926 16:35:40.787708 24303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:35:40.788669 24303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:35:40.789901 24303 hybrid_clock.cc:648] HybridClock initialized: now 1790440540789851 us; error 46 us; skew 500 ppm
I20260926 16:35:40.790253 24303 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:35:40.792716 24303 webserver.cc:460] Webserver started at http://127.6.33.3:45515/ using document root <none> and password file <none>
I20260926 16:35:40.793466 24303 fs_manager.cc:359] Metadata directory not provided
I20260926 16:35:40.793555 24303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:35:40.793838 24303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:35:40.797209 24303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/ts-2/data/instance:
uuid: "81b1304d538b466ab2c8281ef856ee05"
format_stamp: "Formatted at 2026-09-26 16:35:40 on dist-test-slave-khd3"
server_key: "a424e5907ca0ddd826c9d6117449c9d3"
server_key_iv: "dc7190966b0bc9171480f1c5283cdec6"
server_key_version: "encryptionkey@0"
I20260926 16:35:40.798018 24303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/ts-2/wal/instance:
uuid: "81b1304d538b466ab2c8281ef856ee05"
format_stamp: "Formatted at 2026-09-26 16:35:40 on dist-test-slave-khd3"
server_key: "a424e5907ca0ddd826c9d6117449c9d3"
server_key_iv: "dc7190966b0bc9171480f1c5283cdec6"
server_key_version: "encryptionkey@0"
I20260926 16:35:40.803571 24303 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 16:35:40.807262 24317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:35:40.808914 24303 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:35:40.809067 24303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/ts-2/wal
uuid: "81b1304d538b466ab2c8281ef856ee05"
format_stamp: "Formatted at 2026-09-26 16:35:40 on dist-test-slave-khd3"
server_key: "a424e5907ca0ddd826c9d6117449c9d3"
server_key_iv: "dc7190966b0bc9171480f1c5283cdec6"
server_key_version: "encryptionkey@0"
I20260926 16:35:40.809269 24303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:35:40.826517 24303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:35:40.827390 24303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:35:40.827634 24303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:35:40.828665 24303 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:35:40.830282 24303 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:35:40.830374 24303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:35:40.830451 24303 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:35:40.830507 24303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:35:40.866782 24303 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.3:45629
I20260926 16:35:40.866824 24429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.3:45629 every 8 connection(s)
I20260926 16:35:40.868193 24303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
I20260926 16:35:40.875830 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 24303
I20260926 16:35:40.876019 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/ts-2/wal/instance
I20260926 16:35:40.876492 6276 external_mini_cluster.cc:1428] Setting key 8e0ecfba568af7f20ce3fc3b5e63e3f9
I20260926 16:35:40.887418 24171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:40.871447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.3:33391 (local address 127.6.33.62:40795)
0926 16:35:40.871645 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:40.871651 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:40.873365 (+ 1714us) server_negotiation.cc:407] Connection header received
0926 16:35:40.875192 (+ 1827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:40.875198 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:40.875273 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:40.875371 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:35:40.876782 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:40.877724 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:40.879574 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:40.880016 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:40.880156 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:40.886451 (+ 6295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:35:40.886456 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:35:40.886460 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:35:40.886484 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:35:40.886520 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:35:40.886535 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:35:40.886645 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:40.886926 (+ 281us) server_negotiation.cc:299] Negotiation successful
0926 16:35:40.887052 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 16:35:40.888645 24432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:40.871991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:40795 (local address 127.6.33.3:33391)
0926 16:35:40.873161 (+ 1170us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:40.873213 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 16:35:40.874883 (+ 1670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:35:40.875615 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:35:40.875624 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:35:40.875744 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:35:40.876476 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:40.876495 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:40.877891 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:40.877904 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:40.879412 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:40.879426 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:40.880272 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:40.880280 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:40.880528 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:40.885937 (+ 5409us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:35:40.885941 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:35:40.885993 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:35:40.886132 (+ 139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:35:40.886672 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:35:40.886688 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:35:40.886701 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 16:35:40.886911 (+ 210us) client_negotiation.cc:240] Negotiation successful
0926 16:35:40.887422 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":284,"threads_started":1}
I20260926 16:35:40.890724 24430 heartbeater.cc:347] Connected to a master server at 127.6.33.62:40795
I20260926 16:35:40.891109 24430 heartbeater.cc:464] Registering TS with master...
I20260926 16:35:40.892107 24430 heartbeater.cc:511] Master 127.6.33.62:40795 requested a full tablet report, sending...
I20260926 16:35:40.893683 23983 ts_manager.cc:194] Registered new tserver with Master: 81b1304d538b466ab2c8281ef856ee05 (127.6.33.3:45629)
I20260926 16:35:40.894668 23983 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.33.3:33391
I20260926 16:35:40.905318 6276 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:35:40.916390 24171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:40.909768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43598 (local address 127.6.33.62:40795)
0926 16:35:40.909946 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:40.909951 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:40.910364 (+ 413us) server_negotiation.cc:407] Connection header received
0926 16:35:40.910507 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:40.910511 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:40.910590 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:40.910682 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:35:40.911266 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:40.911962 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:40.913676 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:40.914268 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:40.914427 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:40.915690 (+ 1263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:35:40.915695 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:35:40.915698 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:35:40.915720 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:35:40.915753 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:35:40.915772 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:35:40.915853 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:40.916051 (+ 198us) server_negotiation.cc:299] Negotiation successful
0926 16:35:40.916134 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 16:35:40.921108 23983 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43598:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:35:40.930286 23983 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:35:40.949316 24169 heartbeater.cc:503] Master 127.6.33.62:40795 was elected leader, sending a full tablet report...
I20260926 16:35:40.954645 24442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:40.943359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.3:45629 (local address 127.0.0.1:47304)
0926 16:35:40.944318 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:40.944369 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 16:35:40.944593 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:35:40.945594 (+ 1001us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:35:40.945602 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:35:40.945640 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:35:40.946091 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:40.946109 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:40.949164 (+ 3055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:40.949169 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:40.951879 (+ 2710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:40.951891 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:40.953530 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:40.953536 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:40.953821 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:40.953888 (+ 67us) client_negotiation.cc:769] Sending connection context
0926 16:35:40.954058 (+ 170us) client_negotiation.cc:240] Negotiation successful
0926 16:35:40.954276 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"mutex_wait_us":28,"thread_start_us":733,"threads_started":1}
I20260926 16:35:40.954697 24443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:40.943769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47304 (local address 127.6.33.3:45629)
0926 16:35:40.944873 (+ 1104us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:40.944883 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:40.944915 (+ 32us) server_negotiation.cc:407] Connection header received
0926 16:35:40.945014 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:40.945024 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:40.945352 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:40.946342 (+ 990us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:35:40.947366 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:40.949037 (+ 1671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:40.952094 (+ 3057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:40.953306 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:40.953467 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:40.953794 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:40.954137 (+ 343us) server_negotiation.cc:299] Negotiation successful
0926 16:35:40.954329 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":901,"thread_start_us":766,"threads_started":1}
I20260926 16:35:40.956780 24444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:40.944980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:38861 (local address 127.0.0.1:36298)
0926 16:35:40.945594 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:40.945620 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 16:35:40.945782 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:35:40.947796 (+ 2014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:35:40.947802 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:35:40.947828 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:35:40.948337 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:40.948349 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:40.951233 (+ 2884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:40.951239 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:40.953791 (+ 2552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:40.953801 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:40.955874 (+ 2073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:40.955881 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:40.956053 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:40.956088 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 16:35:40.956159 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 16:35:40.956343 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"mutex_wait_us":101,"spinlock_wait_cycles":15104,"thread_start_us":265,"threads_started":1}
I20260926 16:35:40.957278 24446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:40.945594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.2:35419 (local address 127.0.0.1:43724)
0926 16:35:40.946411 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:40.946432 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 16:35:40.946553 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:35:40.947965 (+ 1412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:35:40.947973 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:35:40.948011 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:35:40.948450 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:40.948461 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:40.950519 (+ 2058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:40.950524 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:40.953762 (+ 3238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:40.953782 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:40.956299 (+ 2517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:40.956306 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:40.956506 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:40.956540 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 16:35:40.956627 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 16:35:40.956792 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"mutex_wait_us":70,"spinlock_wait_cycles":896,"thread_start_us":669,"threads_started":1}
I20260926 16:35:40.958127 24445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:40.945221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36298 (local address 127.6.33.1:38861)
0926 16:35:40.947140 (+ 1919us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:40.947162 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:40.947199 (+ 37us) server_negotiation.cc:407] Connection header received
0926 16:35:40.947299 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:40.947307 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:40.947580 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:40.948524 (+ 944us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:35:40.948832 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:40.951062 (+ 2230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:40.954038 (+ 2976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:40.955648 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:40.957183 (+ 1535us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:40.957504 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:40.957615 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 16:35:40.957801 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1678,"thread_start_us":1549,"threads_started":1}
I20260926 16:35:40.958333 24447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:40.946089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43724 (local address 127.6.33.2:35419)
0926 16:35:40.947416 (+ 1327us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:40.947424 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:40.947449 (+ 25us) server_negotiation.cc:407] Connection header received
0926 16:35:40.947547 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:40.947553 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:40.947760 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:40.947984 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:35:40.948628 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:40.950348 (+ 1720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:40.954008 (+ 3660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:40.955769 (+ 1761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:40.955911 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:40.956168 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:40.957702 (+ 1534us) server_negotiation.cc:299] Negotiation successful
0926 16:35:40.957911 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1185,"thread_start_us":1550,"threads_started":1}
I20260926 16:35:40.959542 24365 tablet_service.cc:1467] Processing CreateTablet for tablet cfdcbe25f26c4b00ae3fbf92494a206a (DEFAULT_TABLE table=test-table [id=97fdd9ea95ff4a97936c2856ecb2a728]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:35:40.961344 24365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfdcbe25f26c4b00ae3fbf92494a206a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:35:40.962154 24104 tablet_service.cc:1467] Processing CreateTablet for tablet cfdcbe25f26c4b00ae3fbf92494a206a (DEFAULT_TABLE table=test-table [id=97fdd9ea95ff4a97936c2856ecb2a728]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:35:40.962260 24235 tablet_service.cc:1467] Processing CreateTablet for tablet cfdcbe25f26c4b00ae3fbf92494a206a (DEFAULT_TABLE table=test-table [id=97fdd9ea95ff4a97936c2856ecb2a728]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:35:40.963605 24104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfdcbe25f26c4b00ae3fbf92494a206a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:35:40.963697 24235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfdcbe25f26c4b00ae3fbf92494a206a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:35:40.976889 24448 tablet_bootstrap.cc:492] T cfdcbe25f26c4b00ae3fbf92494a206a P 75c4425b3c0c410795f1ed466357d419: Bootstrap starting.
I20260926 16:35:40.980492 24448 tablet_bootstrap.cc:654] T cfdcbe25f26c4b00ae3fbf92494a206a P 75c4425b3c0c410795f1ed466357d419: Neither blocks nor log segments found. Creating new log.
I20260926 16:35:40.980609 24449 tablet_bootstrap.cc:492] T cfdcbe25f26c4b00ae3fbf92494a206a P 81b1304d538b466ab2c8281ef856ee05: Bootstrap starting.
I20260926 16:35:40.981842 24448 log.cc:824] T cfdcbe25f26c4b00ae3fbf92494a206a P 75c4425b3c0c410795f1ed466357d419: Log is configured to *not* fsync() on all Append() calls
I20260926 16:35:40.982996 24450 tablet_bootstrap.cc:492] T cfdcbe25f26c4b00ae3fbf92494a206a P cde1a2073b514fc8bebc9cc9643ca37d: Bootstrap starting.
I20260926 16:35:40.984414 24449 tablet_bootstrap.cc:654] T cfdcbe25f26c4b00ae3fbf92494a206a P 81b1304d538b466ab2c8281ef856ee05: Neither blocks nor log segments found. Creating new log.
I20260926 16:35:40.985407 24448 tablet_bootstrap.cc:492] T cfdcbe25f26c4b00ae3fbf92494a206a P 75c4425b3c0c410795f1ed466357d419: No bootstrap required, opened a new log
I20260926 16:35:40.985745 24449 log.cc:824] T cfdcbe25f26c4b00ae3fbf92494a206a P 81b1304d538b466ab2c8281ef856ee05: Log is configured to *not* fsync() on all Append() calls
I20260926 16:35:40.985776 24448 ts_tablet_manager.cc:1397] T cfdcbe25f26c4b00ae3fbf92494a206a P 75c4425b3c0c410795f1ed466357d419: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260926 16:35:40.986608 24450 tablet_bootstrap.cc:654] T cfdcbe25f26c4b00ae3fbf92494a206a P cde1a2073b514fc8bebc9cc9643ca37d: Neither blocks nor log segments found. Creating new log.
I20260926 16:35:40.987767 24450 log.cc:824] T cfdcbe25f26c4b00ae3fbf92494a206a P cde1a2073b514fc8bebc9cc9643ca37d: Log is configured to *not* fsync() on all Append() calls
I20260926 16:35:40.988436 24449 tablet_bootstrap.cc:492] T cfdcbe25f26c4b00ae3fbf92494a206a P 81b1304d538b466ab2c8281ef856ee05: No bootstrap required, opened a new log
I20260926 16:35:40.988713 24449 ts_tablet_manager.cc:1397] T cfdcbe25f26c4b00ae3fbf92494a206a P 81b1304d538b466ab2c8281ef856ee05: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260926 16:35:40.990907 24450 tablet_bootstrap.cc:492] T cfdcbe25f26c4b00ae3fbf92494a206a P cde1a2073b514fc8bebc9cc9643ca37d: No bootstrap required, opened a new log
I20260926 16:35:40.991163 24450 ts_tablet_manager.cc:1397] T cfdcbe25f26c4b00ae3fbf92494a206a P cde1a2073b514fc8bebc9cc9643ca37d: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260926 16:35:40.996501 24449 raft_consensus.cc:348] T cfdcbe25f26c4b00ae3fbf92494a206a P 81b1304d538b466ab2c8281ef856ee05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81b1304d538b466ab2c8281ef856ee05" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45629 } } peers { permanent_uuid: "75c4425b3c0c410795f1ed466357d419" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38861 } } peers { permanent_uuid: "cde1a2073b514fc8bebc9cc9643ca37d" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 35419 } }
I20260926 16:35:40.996518 24448 raft_consensus.cc:348] T cfdcbe25f26c4b00ae3fbf92494a206a P 75c4425b3c0c410795f1ed466357d419 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81b1304d538b466ab2c8281ef856ee05" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45629 } } peers { permanent_uuid: "75c4425b3c0c410795f1ed466357d419" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38861 } } peers { permanent_uuid: "cde1a2073b514fc8bebc9cc9643ca37d" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 35419 } }
I20260926 16:35:40.996971 24449 raft_consensus.cc:374] T cfdcbe25f26c4b00ae3fbf92494a206a P 81b1304d538b466ab2c8281ef856ee05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:35:40.996982 24448 raft_consensus.cc:374] T cfdcbe25f26c4b00ae3fbf92494a206a P 75c4425b3c0c410795f1ed466357d419 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:35:40.997071 24448 raft_consensus.cc:729] T cfdcbe25f26c4b00ae3fbf92494a206a P 75c4425b3c0c410795f1ed466357d419 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75c4425b3c0c410795f1ed466357d419, State: Initialized, Role: FOLLOWER
I20260926 16:35:40.997072 24449 raft_consensus.cc:729] T cfdcbe25f26c4b00ae3fbf92494a206a P 81b1304d538b466ab2c8281ef856ee05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81b1304d538b466ab2c8281ef856ee05, State: Initialized, Role: FOLLOWER
I20260926 16:35:40.997751 24449 consensus_queue.cc:260] T cfdcbe25f26c4b00ae3fbf92494a206a P 81b1304d538b466ab2c8281ef856ee05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81b1304d538b466ab2c8281ef856ee05" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45629 } } peers { permanent_uuid: "75c4425b3c0c410795f1ed466357d419" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38861 } } peers { permanent_uuid: "cde1a2073b514fc8bebc9cc9643ca37d" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 35419 } }
I20260926 16:35:40.997782 24448 consensus_queue.cc:260] T cfdcbe25f26c4b00ae3fbf92494a206a P 75c4425b3c0c410795f1ed466357d419 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81b1304d538b466ab2c8281ef856ee05" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45629 } } peers { permanent_uuid: "75c4425b3c0c410795f1ed466357d419" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38861 } } peers { permanent_uuid: "cde1a2073b514fc8bebc9cc9643ca37d" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 35419 } }
I20260926 16:35:40.998873 24430 heartbeater.cc:503] Master 127.6.33.62:40795 was elected leader, sending a full tablet report...
I20260926 16:35:40.999622 24449 ts_tablet_manager.cc:1428] T cfdcbe25f26c4b00ae3fbf92494a206a P 81b1304d538b466ab2c8281ef856ee05: Time spent starting tablet: real 0.011s user 0.006s sys 0.004s
I20260926 16:35:41.003037 24450 raft_consensus.cc:348] T cfdcbe25f26c4b00ae3fbf92494a206a P cde1a2073b514fc8bebc9cc9643ca37d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81b1304d538b466ab2c8281ef856ee05" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45629 } } peers { permanent_uuid: "75c4425b3c0c410795f1ed466357d419" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38861 } } peers { permanent_uuid: "cde1a2073b514fc8bebc9cc9643ca37d" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 35419 } }
I20260926 16:35:41.003489 24450 raft_consensus.cc:374] T cfdcbe25f26c4b00ae3fbf92494a206a P cde1a2073b514fc8bebc9cc9643ca37d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:35:41.003616 24450 raft_consensus.cc:729] T cfdcbe25f26c4b00ae3fbf92494a206a P cde1a2073b514fc8bebc9cc9643ca37d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cde1a2073b514fc8bebc9cc9643ca37d, State: Initialized, Role: FOLLOWER
I20260926 16:35:41.003641 24456 raft_consensus.cc:482] T cfdcbe25f26c4b00ae3fbf92494a206a P 75c4425b3c0c410795f1ed466357d419 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:35:41.003979 24456 raft_consensus.cc:504] T cfdcbe25f26c4b00ae3fbf92494a206a P 75c4425b3c0c410795f1ed466357d419 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81b1304d538b466ab2c8281ef856ee05" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45629 } } peers { permanent_uuid: "75c4425b3c0c410795f1ed466357d419" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38861 } } peers { permanent_uuid: "cde1a2073b514fc8bebc9cc9643ca37d" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 35419 } }
I20260926 16:35:41.004259 24450 consensus_queue.cc:260] T cfdcbe25f26c4b00ae3fbf92494a206a P cde1a2073b514fc8bebc9cc9643ca37d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81b1304d538b466ab2c8281ef856ee05" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45629 } } peers { permanent_uuid: "75c4425b3c0c410795f1ed466357d419" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38861 } } peers { permanent_uuid: "cde1a2073b514fc8bebc9cc9643ca37d" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 35419 } }
I20260926 16:35:41.006197 24456 leader_election.cc:288] T cfdcbe25f26c4b00ae3fbf92494a206a P 75c4425b3c0c410795f1ed466357d419 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 81b1304d538b466ab2c8281ef856ee05 (127.6.33.3:45629), cde1a2073b514fc8bebc9cc9643ca37d (127.6.33.2:35419)
I20260926 16:35:41.012923 24450 ts_tablet_manager.cc:1428] T cfdcbe25f26c4b00ae3fbf92494a206a P cde1a2073b514fc8bebc9cc9643ca37d: Time spent starting tablet: real 0.022s user 0.017s sys 0.006s
I20260926 16:35:41.013365 24300 heartbeater.cc:503] Master 127.6.33.62:40795 was elected leader, sending a full tablet report...
I20260926 16:35:41.019315 24448 ts_tablet_manager.cc:1428] T cfdcbe25f26c4b00ae3fbf92494a206a P 75c4425b3c0c410795f1ed466357d419: Time spent starting tablet: real 0.033s user 0.021s sys 0.004s
I20260926 16:35:41.021171 24443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:41.009121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:36309 (local address 127.6.33.3:45629)
0926 16:35:41.009516 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:41.009525 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:41.013255 (+ 3730us) server_negotiation.cc:407] Connection header received
0926 16:35:41.013774 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:41.013780 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:41.013856 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:41.013919 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:35:41.014745 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:41.015789 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:41.018611 (+ 2822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:41.019883 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:41.019984 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:41.020104 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:41.020748 (+ 644us) server_negotiation.cc:299] Negotiation successful
0926 16:35:41.020911 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229}
I20260926 16:35:41.021205 24458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:41.006433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.3:45629 (local address 127.6.33.1:36309)
0926 16:35:41.012995 (+ 6562us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:41.013155 (+ 160us) client_negotiation.cc:174] Beginning negotiation
0926 16:35:41.013434 (+ 279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:35:41.014121 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:35:41.014126 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:35:41.014152 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:35:41.014572 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:41.014586 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:41.015988 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:41.015994 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:41.018385 (+ 2391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:41.018401 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:41.020362 (+ 1961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:41.020369 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:41.020559 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:41.020595 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 16:35:41.020695 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 16:35:41.020864 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6395,"thread_start_us":6158,"threads_started":1}
I20260926 16:35:41.022533 24385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfdcbe25f26c4b00ae3fbf92494a206a" candidate_uuid: "75c4425b3c0c410795f1ed466357d419" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81b1304d538b466ab2c8281ef856ee05" is_pre_election: true
I20260926 16:35:41.023062 24385 raft_consensus.cc:2454] T cfdcbe25f26c4b00ae3fbf92494a206a P 81b1304d538b466ab2c8281ef856ee05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75c4425b3c0c410795f1ed466357d419 in term 0.
I20260926 16:35:41.023782 24058 leader_election.cc:302] T cfdcbe25f26c4b00ae3fbf92494a206a P 75c4425b3c0c410795f1ed466357d419 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75c4425b3c0c410795f1ed466357d419, 81b1304d538b466ab2c8281ef856ee05; no voters:
I20260926 16:35:41.024235 24455 raft_consensus.cc:2781] T cfdcbe25f26c4b00ae3fbf92494a206a P 75c4425b3c0c410795f1ed466357d419 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:35:41.024401 24455 raft_consensus.cc:482] T cfdcbe25f26c4b00ae3fbf92494a206a P 75c4425b3c0c410795f1ed466357d419 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:35:41.024547 24455 raft_consensus.cc:3037] T cfdcbe25f26c4b00ae3fbf92494a206a P 75c4425b3c0c410795f1ed466357d419 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:35:41.025839 24459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:41.013805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.2:35419 (local address 127.6.33.1:32791)
0926 16:35:41.019153 (+ 5348us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:41.019172 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 16:35:41.019305 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:35:41.019798 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:35:41.019801 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:35:41.019822 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:35:41.020213 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:41.020229 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:41.021679 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:41.021687 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:41.023350 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:41.023365 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:41.024959 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:41.024965 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:41.025175 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:41.025221 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 16:35:41.025331 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 16:35:41.025502 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5203,"thread_start_us":5029,"threads_started":1}
I20260926 16:35:41.026083 24447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:41.013576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:32791 (local address 127.6.33.2:35419)
0926 16:35:41.014860 (+ 1284us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:41.014866 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:41.019277 (+ 4411us) server_negotiation.cc:407] Connection header received
0926 16:35:41.019448 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:41.019454 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:41.019552 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:41.019652 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:35:41.020411 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:41.021472 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:41.023576 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:41.024707 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:41.024891 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:41.024970 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:41.025656 (+ 686us) server_negotiation.cc:299] Negotiation successful
0926 16:35:41.025807 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1134}
I20260926 16:35:41.027130 24255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfdcbe25f26c4b00ae3fbf92494a206a" candidate_uuid: "75c4425b3c0c410795f1ed466357d419" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cde1a2073b514fc8bebc9cc9643ca37d" is_pre_election: true
I20260926 16:35:41.027674 24255 raft_consensus.cc:2454] T cfdcbe25f26c4b00ae3fbf92494a206a P cde1a2073b514fc8bebc9cc9643ca37d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75c4425b3c0c410795f1ed466357d419 in term 0.
I20260926 16:35:41.027828 24455 raft_consensus.cc:504] T cfdcbe25f26c4b00ae3fbf92494a206a P 75c4425b3c0c410795f1ed466357d419 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81b1304d538b466ab2c8281ef856ee05" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45629 } } peers { permanent_uuid: "75c4425b3c0c410795f1ed466357d419" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38861 } } peers { permanent_uuid: "cde1a2073b514fc8bebc9cc9643ca37d" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 35419 } }
I20260926 16:35:41.028570 24455 leader_election.cc:288] T cfdcbe25f26c4b00ae3fbf92494a206a P 75c4425b3c0c410795f1ed466357d419 [CANDIDATE]: Term 1 election: Requested vote from peers 81b1304d538b466ab2c8281ef856ee05 (127.6.33.3:45629), cde1a2073b514fc8bebc9cc9643ca37d (127.6.33.2:35419)
I20260926 16:35:41.029110 24385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfdcbe25f26c4b00ae3fbf92494a206a" candidate_uuid: "75c4425b3c0c410795f1ed466357d419" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81b1304d538b466ab2c8281ef856ee05"
I20260926 16:35:41.029136 24255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfdcbe25f26c4b00ae3fbf92494a206a" candidate_uuid: "75c4425b3c0c410795f1ed466357d419" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cde1a2073b514fc8bebc9cc9643ca37d"
I20260926 16:35:41.029335 24385 raft_consensus.cc:3037] T cfdcbe25f26c4b00ae3fbf92494a206a P 81b1304d538b466ab2c8281ef856ee05 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:35:41.029341 24255 raft_consensus.cc:3037] T cfdcbe25f26c4b00ae3fbf92494a206a P cde1a2073b514fc8bebc9cc9643ca37d [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:35:41.031689 24255 raft_consensus.cc:2454] T cfdcbe25f26c4b00ae3fbf92494a206a P cde1a2073b514fc8bebc9cc9643ca37d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75c4425b3c0c410795f1ed466357d419 in term 1.
I20260926 16:35:41.031883 24385 raft_consensus.cc:2454] T cfdcbe25f26c4b00ae3fbf92494a206a P 81b1304d538b466ab2c8281ef856ee05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75c4425b3c0c410795f1ed466357d419 in term 1.
I20260926 16:35:41.032166 24058 leader_election.cc:302] T cfdcbe25f26c4b00ae3fbf92494a206a P 75c4425b3c0c410795f1ed466357d419 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75c4425b3c0c410795f1ed466357d419, cde1a2073b514fc8bebc9cc9643ca37d; no voters:
I20260926 16:35:41.032506 24455 raft_consensus.cc:2781] T cfdcbe25f26c4b00ae3fbf92494a206a P 75c4425b3c0c410795f1ed466357d419 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:35:41.032687 24455 raft_consensus.cc:686] T cfdcbe25f26c4b00ae3fbf92494a206a P 75c4425b3c0c410795f1ed466357d419 [term 1 LEADER]: Becoming Leader. State: Replica: 75c4425b3c0c410795f1ed466357d419, State: Running, Role: LEADER
I20260926 16:35:41.033170 24455 consensus_queue.cc:237] T cfdcbe25f26c4b00ae3fbf92494a206a P 75c4425b3c0c410795f1ed466357d419 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81b1304d538b466ab2c8281ef856ee05" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45629 } } peers { permanent_uuid: "75c4425b3c0c410795f1ed466357d419" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38861 } } peers { permanent_uuid: "cde1a2073b514fc8bebc9cc9643ca37d" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 35419 } }
I20260926 16:35:41.037683 23983 catalog_manager.cc:5495] T cfdcbe25f26c4b00ae3fbf92494a206a P 75c4425b3c0c410795f1ed466357d419 reported cstate change: term changed from 0 to 1, leader changed from <none> to 75c4425b3c0c410795f1ed466357d419 (127.6.33.1). New cstate: current_term: 1 leader_uuid: "75c4425b3c0c410795f1ed466357d419" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81b1304d538b466ab2c8281ef856ee05" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75c4425b3c0c410795f1ed466357d419" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 38861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cde1a2073b514fc8bebc9cc9643ca37d" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 35419 } health_report { overall_health: UNKNOWN } } }
I20260926 16:35:41.058802 24445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:41.053273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36300 (local address 127.6.33.1:38861)
0926 16:35:41.053487 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:41.053493 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:41.053513 (+ 20us) server_negotiation.cc:407] Connection header received
0926 16:35:41.053768 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:41.053772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:41.053852 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:41.054009 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:35:41.054446 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:41.055162 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:41.056521 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:41.057044 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:41.057176 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:41.057678 (+ 502us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:35:41.058234 (+ 556us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:35:41.058400 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:41.058460 (+ 60us) server_negotiation.cc:299] Negotiation successful
0926 16:35:41.058541 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 16:35:41.067633 24385 raft_consensus.cc:1261] T cfdcbe25f26c4b00ae3fbf92494a206a P 81b1304d538b466ab2c8281ef856ee05 [term 1 FOLLOWER]: Refusing update from remote peer 75c4425b3c0c410795f1ed466357d419: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:35:41.067626 24255 raft_consensus.cc:1261] T cfdcbe25f26c4b00ae3fbf92494a206a P cde1a2073b514fc8bebc9cc9643ca37d [term 1 FOLLOWER]: Refusing update from remote peer 75c4425b3c0c410795f1ed466357d419: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:35:41.068631 24456 consensus_queue.cc:1035] T cfdcbe25f26c4b00ae3fbf92494a206a P 75c4425b3c0c410795f1ed466357d419 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81b1304d538b466ab2c8281ef856ee05" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 45629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:35:41.069026 24455 consensus_queue.cc:1035] T cfdcbe25f26c4b00ae3fbf92494a206a P 75c4425b3c0c410795f1ed466357d419 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cde1a2073b514fc8bebc9cc9643ca37d" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 35419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:35:41.081362 24462 mvcc.cc:205] Tried to move back new op lower bound from 7333644456201809920 to 7333644456075853824. Current Snapshot: MvccSnapshot[applied={T|T < 7333644456201809920}]
I20260926 16:35:41.081864 24463 mvcc.cc:205] Tried to move back new op lower bound from 7333644456201809920 to 7333644456075853824. Current Snapshot: MvccSnapshot[applied={T|T < 7333644456201809920}]
I20260926 16:35:41.086129 24464 mvcc.cc:205] Tried to move back new op lower bound from 7333644456201809920 to 7333644456075853824. Current Snapshot: MvccSnapshot[applied={T|T < 7333644456201809920}]
I20260926 16:35:41.095552 23982 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43598:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:35:41.095858 23982 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43598:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:35:41.099835 23982 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 81fef6a4e57045b8802a7f9b0c327ee6: Sending DeleteTablet for 3 replicas of tablet cfdcbe25f26c4b00ae3fbf92494a206a
I20260926 16:35:41.101078 24365 tablet_service.cc:1514] Processing DeleteTablet for tablet cfdcbe25f26c4b00ae3fbf92494a206a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:35:41 UTC) from {username='slave'} at 127.0.0.1:47304
I20260926 16:35:41.101269 24235 tablet_service.cc:1514] Processing DeleteTablet for tablet cfdcbe25f26c4b00ae3fbf92494a206a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:35:41 UTC) from {username='slave'} at 127.0.0.1:43724
I20260926 16:35:41.101595 24104 tablet_service.cc:1514] Processing DeleteTablet for tablet cfdcbe25f26c4b00ae3fbf92494a206a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:35:41 UTC) from {username='slave'} at 127.0.0.1:36298
I20260926 16:35:41.103579 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 24042
I20260926 16:35:41.113708 24471 tablet_replica.cc:330] T cfdcbe25f26c4b00ae3fbf92494a206a P 81b1304d538b466ab2c8281ef856ee05: stopping tablet replica
I20260926 16:35:41.114420 24471 raft_consensus.cc:2229] T cfdcbe25f26c4b00ae3fbf92494a206a P 81b1304d538b466ab2c8281ef856ee05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:35:41.115111 24471 raft_consensus.cc:2258] T cfdcbe25f26c4b00ae3fbf92494a206a P 81b1304d538b466ab2c8281ef856ee05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:35:41.117894 24471 ts_tablet_manager.cc:1905] T cfdcbe25f26c4b00ae3fbf92494a206a P 81b1304d538b466ab2c8281ef856ee05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:35:41.125878 24471 ts_tablet_manager.cc:1918] T cfdcbe25f26c4b00ae3fbf92494a206a P 81b1304d538b466ab2c8281ef856ee05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:35:41.126219 24471 log.cc:1196] T cfdcbe25f26c4b00ae3fbf92494a206a P 81b1304d538b466ab2c8281ef856ee05: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/ts-2/wal/wals/cfdcbe25f26c4b00ae3fbf92494a206a
I20260926 16:35:41.126916 24471 ts_tablet_manager.cc:1939] T cfdcbe25f26c4b00ae3fbf92494a206a P 81b1304d538b466ab2c8281ef856ee05: Deleting consensus metadata
I20260926 16:35:41.129269 23968 catalog_manager.cc:4841] TS 81b1304d538b466ab2c8281ef856ee05 (127.6.33.3:45629): tablet cfdcbe25f26c4b00ae3fbf92494a206a (table test-table [id=97fdd9ea95ff4a97936c2856ecb2a728]) successfully deleted
I20260926 16:35:41.130023 24473 tablet_replica.cc:330] T cfdcbe25f26c4b00ae3fbf92494a206a P 75c4425b3c0c410795f1ed466357d419: stopping tablet replica
I20260926 16:35:41.130630 24473 raft_consensus.cc:2229] T cfdcbe25f26c4b00ae3fbf92494a206a P 75c4425b3c0c410795f1ed466357d419 [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:35:41.130944 24472 tablet_replica.cc:330] T cfdcbe25f26c4b00ae3fbf92494a206a P cde1a2073b514fc8bebc9cc9643ca37d: stopping tablet replica
I20260926 16:35:41.131357 24473 raft_consensus.cc:2258] T cfdcbe25f26c4b00ae3fbf92494a206a P 75c4425b3c0c410795f1ed466357d419 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:35:41.131413 24472 raft_consensus.cc:2229] T cfdcbe25f26c4b00ae3fbf92494a206a P cde1a2073b514fc8bebc9cc9643ca37d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:35:41.131958 24472 raft_consensus.cc:2258] T cfdcbe25f26c4b00ae3fbf92494a206a P cde1a2073b514fc8bebc9cc9643ca37d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:35:41.134265 24473 ts_tablet_manager.cc:1905] T cfdcbe25f26c4b00ae3fbf92494a206a P 75c4425b3c0c410795f1ed466357d419: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:35:41.134258 24472 ts_tablet_manager.cc:1905] T cfdcbe25f26c4b00ae3fbf92494a206a P cde1a2073b514fc8bebc9cc9643ca37d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:35:41.137661 24445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:41.104896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36314 (local address 127.6.33.1:38861)
0926 16:35:41.105110 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:41.105117 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:41.127695 (+ 22578us) server_negotiation.cc:407] Connection header received
0926 16:35:41.128321 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:41.128331 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:41.128486 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:41.128582 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:35:41.129473 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:41.130785 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:41.133253 (+ 2468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:41.133922 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:41.134030 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:41.135726 (+ 1696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:35:41.135738 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:35:41.135756 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:35:41.136628 (+ 872us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:35:41.136710 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:35:41.136742 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:35:41.136876 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:41.137134 (+ 258us) server_negotiation.cc:299] Negotiation successful
0926 16:35:41.137249 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 16:35:41.139227 24164 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:35:41.141534 24472 ts_tablet_manager.cc:1918] T cfdcbe25f26c4b00ae3fbf92494a206a P cde1a2073b514fc8bebc9cc9643ca37d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:35:41.141741 24472 log.cc:1196] T cfdcbe25f26c4b00ae3fbf92494a206a P cde1a2073b514fc8bebc9cc9643ca37d: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440233850748-6276-0/minicluster-data/ts-1/wal/wals/cfdcbe25f26c4b00ae3fbf92494a206a
I20260926 16:35:41.142321 24472 ts_tablet_manager.cc:1939] T cfdcbe25f26c4b00ae3fbf92494a206a P cde1a2073b514fc8bebc9cc9643ca37d: Deleting consensus metadata
I20260926 16:35:41.144294 23968 catalog_manager.cc:4841] TS cde1a2073b514fc8bebc9cc9643ca37d (127.6.33.2:35419): tablet cfdcbe25f26c4b00ae3fbf92494a206a (table test-table [id=97fdd9ea95ff4a97936c2856ecb2a728]) successfully deleted
I20260926 16:35:41.241102 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 24042
W20260926 16:35:41.259790 23968 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.33.1:38861 (error 108)
I20260926 16:35:41.260572 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 24173
I20260926 16:35:41.261718 24446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:41.261254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:38861 (local address 127.0.0.1:36324)
0926 16:35:41.261445 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:41.261574 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.1:38861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260926 16:35:41.261961 23968 catalog_manager.cc:4584] TS 75c4425b3c0c410795f1ed466357d419 (127.6.33.1:38861): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cfdcbe25f26c4b00ae3fbf92494a206a: Network error: Client connection negotiation failed: client connection to 127.6.33.1:38861: connect: Connection refused (error 111)
I20260926 16:35:41.267861 24447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:41.261430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43738 (local address 127.6.33.2:35419)
0926 16:35:41.261617 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:41.261625 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:41.261645 (+ 20us) server_negotiation.cc:407] Connection header received
0926 16:35:41.261899 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:41.261905 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:41.262025 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:41.262118 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:35:41.262656 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:41.263410 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:41.264742 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:41.265363 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:41.265461 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:41.266504 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:35:41.266513 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:35:41.266527 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:35:41.267126 (+ 599us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:35:41.267173 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:35:41.267192 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:35:41.267280 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:41.267476 (+ 196us) server_negotiation.cc:299] Negotiation successful
0926 16:35:41.267597 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 16:35:41.268532 24295 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:35:41.315696 24446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:41.315203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:38861 (local address 127.0.0.1:36336)
0926 16:35:41.315419 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:41.315512 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.1:38861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260926 16:35:41.317066 24446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:41.316477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:38861 (local address 127.0.0.1:36342)
0926 16:35:41.316805 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:41.316930 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.1:38861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158}
I20260926 16:35:41.359710 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 24173
I20260926 16:35:41.376528 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 24303
I20260926 16:35:41.378291 24446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:41.377807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:38861 (local address 127.0.0.1:36344)
0926 16:35:41.377997 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:41.378080 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.1:38861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260926 16:35:41.379212 24446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:41.378891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:38861 (local address 127.0.0.1:36352)
0926 16:35:41.379041 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:41.379112 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.1:38861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 16:35:41.383832 24443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:41.377484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47320 (local address 127.6.33.3:45629)
0926 16:35:41.377677 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:41.377684 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:41.377715 (+ 31us) server_negotiation.cc:407] Connection header received
0926 16:35:41.377886 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:41.377895 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:41.377982 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:41.378098 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:35:41.378576 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:41.379342 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:41.380691 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:41.381178 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:41.381310 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:41.382408 (+ 1098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:35:41.382426 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:35:41.382441 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:35:41.383026 (+ 585us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:35:41.383097 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:35:41.383148 (+ 51us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:35:41.383258 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:41.383406 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 16:35:41.383523 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 16:35:41.384492 24425 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:35:41.459975 24446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:41.459445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:38861 (local address 127.0.0.1:36364)
0926 16:35:41.459697 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:41.459803 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.1:38861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260926 16:35:41.461308 24446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:41.460896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:38861 (local address 127.0.0.1:36372)
0926 16:35:41.461122 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:41.461194 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.1:38861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260926 16:35:41.475740 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 24303
I20260926 16:35:41.492282 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 23952
I20260926 16:35:41.493521 24013 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:35:41.602599 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 23952
2026-09-26T16:35:41Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2783 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:35:41 dist-test-slave-khd3 krb5kdc[24482](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:35:41 dist-test-slave-khd3 krb5kdc[24482](info): set up 2 sockets
Sep 26 16:35:41 dist-test-slave-khd3 krb5kdc[24482](info): commencing operation
krb5kdc: starting...
W20260926 16:35:42.725956 6276 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Sep 26 16:35:42 dist-test-slave-khd3 krb5kdc[24482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:35:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:35:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab.
Entry for principal kudu/127.6.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab.
I20260926 16:35:42.979550 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:34389
--webserver_interface=127.6.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:45679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.33.62:34389
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:35:43.322038 24495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:35:43.322348 24495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:35:43.322515 24495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:35:43.329882 24495 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:35:43.329994 24495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:35:43.330176 24495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:35:43.330288 24495 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:35:43.330394 24495 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:35:43.340648 24495 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:45679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.33.62:34389
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:34389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:35:43.342139 24495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:35:43.343546 24495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:35:43.355603 24500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:35:43.355571 24503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:35:43.355647 24501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:35:43.356794 24495 server_base.cc:1034] running on GCE node
I20260926 16:35:43.357662 24495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:35:43.359107 24495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:35:43.360321 24495 hybrid_clock.cc:648] HybridClock initialized: now 1790440543360267 us; error 43 us; skew 500 ppm
I20260926 16:35:43.360734 24495 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:35:43 dist-test-slave-khd3 krb5kdc[24482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440543, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:35:43.364118 24495 init.cc:377] Logged in from keytab as kudu/127.6.33.62@KRBTEST.COM (short username kudu)
I20260926 16:35:43.365756 24495 webserver.cc:460] Webserver started at http://127.6.33.62:40199/ using document root <none> and password file <none>
I20260926 16:35:43.366477 24495 fs_manager.cc:359] Metadata directory not provided
I20260926 16:35:43.366638 24495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:35:43.366946 24495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:35:43.370101 24495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/master-0/data/instance:
uuid: "2e2480dddece4a42a57d58e5f3928b8b"
format_stamp: "Formatted at 2026-09-26 16:35:43 on dist-test-slave-khd3"
server_key: "f71feb7ff546b9708e51c1cca0689f22"
server_key_iv: "ce09cf833c8467131be0c1e1d12f6465"
server_key_version: "encryptionkey@0"
I20260926 16:35:43.370880 24495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/master-0/wal/instance:
uuid: "2e2480dddece4a42a57d58e5f3928b8b"
format_stamp: "Formatted at 2026-09-26 16:35:43 on dist-test-slave-khd3"
server_key: "f71feb7ff546b9708e51c1cca0689f22"
server_key_iv: "ce09cf833c8467131be0c1e1d12f6465"
server_key_version: "encryptionkey@0"
I20260926 16:35:43.376370 24495 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 16:35:43.380339 24510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:35:43.381968 24495 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 16:35:43.382205 24495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/master-0/wal
uuid: "2e2480dddece4a42a57d58e5f3928b8b"
format_stamp: "Formatted at 2026-09-26 16:35:43 on dist-test-slave-khd3"
server_key: "f71feb7ff546b9708e51c1cca0689f22"
server_key_iv: "ce09cf833c8467131be0c1e1d12f6465"
server_key_version: "encryptionkey@0"
I20260926 16:35:43.382431 24495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:35:43.403082 24495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:35:43.407743 24495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:35:43.408036 24495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:35:43.427311 24495 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.62:34389
I20260926 16:35:43.427349 24561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.62:34389 every 8 connection(s)
I20260926 16:35:43.428819 24495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
I20260926 16:35:43.432672 24562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:35:43.436275 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 24495
I20260926 16:35:43.436461 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/master-0/wal/instance
I20260926 16:35:43.437074 6276 external_mini_cluster.cc:1428] Setting key dd35c155df6c935aa47bebe68a42b508
I20260926 16:35:43.446403 24562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e2480dddece4a42a57d58e5f3928b8b: Bootstrap starting.
I20260926 16:35:43.450593 24562 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e2480dddece4a42a57d58e5f3928b8b: Neither blocks nor log segments found. Creating new log.
I20260926 16:35:43.452029 24562 log.cc:824] T 00000000000000000000000000000000 P 2e2480dddece4a42a57d58e5f3928b8b: Log is configured to *not* fsync() on all Append() calls
Sep 26 16:35:43 dist-test-slave-khd3 krb5kdc[24482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:35:43.455662 24562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e2480dddece4a42a57d58e5f3928b8b: No bootstrap required, opened a new log
I20260926 16:35:43.463852 24565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:43.439133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46174 (local address 127.6.33.62:34389)
0926 16:35:43.439998 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:43.440010 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:43.440056 (+ 46us) server_negotiation.cc:407] Connection header received
0926 16:35:43.441931 (+ 1875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:43.441958 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:43.442435 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:43.442820 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:35:43.444751 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:43.446364 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:43.448328 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:43.449162 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:43.449315 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:43.454552 (+ 5237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:35:43.454563 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:35:43.454584 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:35:43.454635 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:35:43.458877 (+ 4242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:35:43.459702 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:35:43.459717 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:35:43.459727 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:35:43.459946 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:35:43.460351 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:35:43.460357 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:35:43.460360 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:35:43.461145 (+ 785us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:35:43.461453 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:43.461891 (+ 438us) server_negotiation.cc:299] Negotiation successful
0926 16:35:43.462375 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":401,"threads_started":1}
I20260926 16:35:43.466621 24562 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e2480dddece4a42a57d58e5f3928b8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e2480dddece4a42a57d58e5f3928b8b" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 34389 } }
I20260926 16:35:43.467070 24562 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e2480dddece4a42a57d58e5f3928b8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:35:43.467185 24562 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e2480dddece4a42a57d58e5f3928b8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e2480dddece4a42a57d58e5f3928b8b, State: Initialized, Role: FOLLOWER
I20260926 16:35:43.467770 24562 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e2480dddece4a42a57d58e5f3928b8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e2480dddece4a42a57d58e5f3928b8b" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 34389 } }
I20260926 16:35:43.468032 24562 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e2480dddece4a42a57d58e5f3928b8b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:35:43.468186 24562 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e2480dddece4a42a57d58e5f3928b8b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:35:43.468360 24562 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e2480dddece4a42a57d58e5f3928b8b [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:35:43.470861 24562 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e2480dddece4a42a57d58e5f3928b8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e2480dddece4a42a57d58e5f3928b8b" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 34389 } }
I20260926 16:35:43.471372 24562 leader_election.cc:302] T 00000000000000000000000000000000 P 2e2480dddece4a42a57d58e5f3928b8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e2480dddece4a42a57d58e5f3928b8b; no voters:
I20260926 16:35:43.471977 24562 leader_election.cc:288] T 00000000000000000000000000000000 P 2e2480dddece4a42a57d58e5f3928b8b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:35:43.472167 24567 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e2480dddece4a42a57d58e5f3928b8b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:35:43.472957 24567 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e2480dddece4a42a57d58e5f3928b8b [term 1 LEADER]: Becoming Leader. State: Replica: 2e2480dddece4a42a57d58e5f3928b8b, State: Running, Role: LEADER
I20260926 16:35:43.473541 24567 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e2480dddece4a42a57d58e5f3928b8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e2480dddece4a42a57d58e5f3928b8b" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 34389 } }
I20260926 16:35:43.474180 24562 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e2480dddece4a42a57d58e5f3928b8b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:35:43.476826 24568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e2480dddece4a42a57d58e5f3928b8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e2480dddece4a42a57d58e5f3928b8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e2480dddece4a42a57d58e5f3928b8b" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 34389 } } }
I20260926 16:35:43.477300 24568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e2480dddece4a42a57d58e5f3928b8b [sys.catalog]: This master's current role is: LEADER
I20260926 16:35:43.476805 24569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e2480dddece4a42a57d58e5f3928b8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e2480dddece4a42a57d58e5f3928b8b. Latest consensus state: current_term: 1 leader_uuid: "2e2480dddece4a42a57d58e5f3928b8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e2480dddece4a42a57d58e5f3928b8b" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 34389 } } }
I20260926 16:35:43.477880 24569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e2480dddece4a42a57d58e5f3928b8b [sys.catalog]: This master's current role is: LEADER
I20260926 16:35:43.479645 24573 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:35:43.485561 24573 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:35:43.495285 24573 catalog_manager.cc:1347] Generated new cluster ID: 2546d3af1ab54bd3829482a871158b29
I20260926 16:35:43.495409 24573 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:35:43.537896 24573 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:35:43.540155 24573 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:35:43.549821 24573 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e2480dddece4a42a57d58e5f3928b8b: Generated new TSK 0
I20260926 16:35:43.551285 24573 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.33.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab.
Entry for principal kudu/127.6.33.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab.
I20260926 16:35:43.627640 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.1:0
--local_ip_for_outbound_sockets=127.6.33.1
--webserver_interface=127.6.33.1
--webserver_port=0
--tserver_master_addrs=127.6.33.62:34389
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab
--principal=kudu/127.6.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:45679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:35:43.987550 24588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:35:43.987846 24588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:35:43.987943 24588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:35:43.996642 24588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:35:43.996795 24588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:35:43.996973 24588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.1
I20260926 16:35:44.007280 24588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:45679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab
--principal=kudu/127.6.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.33.1
--webserver_port=0
--tserver_master_addrs=127.6.33.62:34389
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.1
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:35:44.008570 24588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:35:44.009948 24588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:35:44.022686 24593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:35:44.022773 24596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:35:44.022782 24594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:35:44.023838 24588 server_base.cc:1034] running on GCE node
I20260926 16:35:44.024433 24588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:35:44.025384 24588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:35:44.026600 24588 hybrid_clock.cc:648] HybridClock initialized: now 1790440544026536 us; error 55 us; skew 500 ppm
I20260926 16:35:44.026902 24588 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:35:44 dist-test-slave-khd3 krb5kdc[24482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440544, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:35:44.030184 24588 init.cc:377] Logged in from keytab as kudu/127.6.33.1@KRBTEST.COM (short username kudu)
I20260926 16:35:44.031723 24588 webserver.cc:460] Webserver started at http://127.6.33.1:43235/ using document root <none> and password file <none>
I20260926 16:35:44.032423 24588 fs_manager.cc:359] Metadata directory not provided
I20260926 16:35:44.032554 24588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:35:44.032955 24588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:35:44.035956 24588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/ts-0/data/instance:
uuid: "f9ae241ae3694966bb900e5546bc2fb9"
format_stamp: "Formatted at 2026-09-26 16:35:44 on dist-test-slave-khd3"
server_key: "4ab6feec888fbf8c9308f3e83a0c0497"
server_key_iv: "9f3b842e7c350807528c5c82f2c69db2"
server_key_version: "encryptionkey@0"
I20260926 16:35:44.036962 24588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/ts-0/wal/instance:
uuid: "f9ae241ae3694966bb900e5546bc2fb9"
format_stamp: "Formatted at 2026-09-26 16:35:44 on dist-test-slave-khd3"
server_key: "4ab6feec888fbf8c9308f3e83a0c0497"
server_key_iv: "9f3b842e7c350807528c5c82f2c69db2"
server_key_version: "encryptionkey@0"
I20260926 16:35:44.042706 24588 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 16:35:44.046893 24603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:35:44.048815 24588 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260926 16:35:44.049062 24588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/ts-0/wal
uuid: "f9ae241ae3694966bb900e5546bc2fb9"
format_stamp: "Formatted at 2026-09-26 16:35:44 on dist-test-slave-khd3"
server_key: "4ab6feec888fbf8c9308f3e83a0c0497"
server_key_iv: "9f3b842e7c350807528c5c82f2c69db2"
server_key_version: "encryptionkey@0"
I20260926 16:35:44.049283 24588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:35:44.063716 24588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:35:44.067883 24588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:35:44.068140 24588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:35:44.069137 24588 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:35:44.070420 24588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:35:44.070492 24588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:35:44.070585 24588 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:35:44.070631 24588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:35:44.105249 24588 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.1:35595
I20260926 16:35:44.105304 24715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.1:35595 every 8 connection(s)
I20260926 16:35:44.106557 24588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
I20260926 16:35:44.108014 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 24588
I20260926 16:35:44.108217 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/ts-0/wal/instance
I20260926 16:35:44.108814 6276 external_mini_cluster.cc:1428] Setting key 609cd4c6a2a595a6b922d9c210262ebd
Sep 26 16:35:44 dist-test-slave-khd3 krb5kdc[24482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440544, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.1@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:35:44.128870 24718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:44.109514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:40497 (local address 127.6.33.62:34389)
0926 16:35:44.110026 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:44.110033 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:44.111084 (+ 1051us) server_negotiation.cc:407] Connection header received
0926 16:35:44.113009 (+ 1925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:44.113014 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:44.113106 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:44.113242 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:35:44.114826 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:44.115547 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:44.117213 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:44.117810 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:44.117961 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:44.121707 (+ 3746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:35:44.121716 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:35:44.121725 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:35:44.121769 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:35:44.125438 (+ 3669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:35:44.126156 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:35:44.126164 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:35:44.126171 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:35:44.126329 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:35:44.126792 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:35:44.126800 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:35:44.126806 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:35:44.127187 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:35:44.127342 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:44.128074 (+ 732us) server_negotiation.cc:299] Negotiation successful
0926 16:35:44.128336 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":906,"threads_started":1}
I20260926 16:35:44.129573 24719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:44.110094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:34389 (local address 127.6.33.1:40497)
0926 16:35:44.110932 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:44.110975 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 16:35:44.112671 (+ 1696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:35:44.113483 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:35:44.113492 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:35:44.114053 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:35:44.114639 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:44.114658 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:44.115719 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:44.115725 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:44.117064 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:44.117077 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:44.118089 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:44.118095 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:44.118299 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:44.119025 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:35:44.119048 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:35:44.121451 (+ 2403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:35:44.125578 (+ 4127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:35:44.125584 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:35:44.125596 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:35:44.126010 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:35:44.126443 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:35:44.126446 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:35:44.126448 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:35:44.126638 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:35:44.127335 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:35:44.127341 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:35:44.127675 (+ 334us) client_negotiation.cc:769] Sending connection context
0926 16:35:44.127901 (+ 226us) client_negotiation.cc:240] Negotiation successful
0926 16:35:44.128295 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":319,"threads_started":1}
I20260926 16:35:44.131325 24716 heartbeater.cc:347] Connected to a master server at 127.6.33.62:34389
I20260926 16:35:44.131722 24716 heartbeater.cc:464] Registering TS with master...
I20260926 16:35:44.132701 24716 heartbeater.cc:511] Master 127.6.33.62:34389 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.33.2@KRBTEST.COM; defaulting to no policy
I20260926 16:35:44.135445 24527 ts_manager.cc:194] Registered new tserver with Master: f9ae241ae3694966bb900e5546bc2fb9 (127.6.33.1:35595)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.2@KRBTEST.COM" created.
I20260926 16:35:44.137322 24527 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.33.1@KRBTEST.COM'} at 127.6.33.1:40497
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab.
Entry for principal kudu/127.6.33.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab.
I20260926 16:35:44.168262 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.2:0
--local_ip_for_outbound_sockets=127.6.33.2
--webserver_interface=127.6.33.2
--webserver_port=0
--tserver_master_addrs=127.6.33.62:34389
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab
--principal=kudu/127.6.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:45679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:35:44.502508 24722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:35:44.502801 24722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:35:44.502955 24722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:35:44.510309 24722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:35:44.510430 24722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:35:44.510638 24722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.2
I20260926 16:35:44.520576 24722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:45679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab
--principal=kudu/127.6.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.33.2
--webserver_port=0
--tserver_master_addrs=127.6.33.62:34389
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.2
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:35:44.521900 24722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:35:44.523233 24722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:35:44.536872 24727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:35:44.536907 24728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:35:44.536967 24730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:35:44.543224 24722 server_base.cc:1034] running on GCE node
I20260926 16:35:44.544015 24722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:35:44.545461 24722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:35:44.546751 24722 hybrid_clock.cc:648] HybridClock initialized: now 1790440544546689 us; error 69 us; skew 500 ppm
I20260926 16:35:44.547127 24722 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:35:44 dist-test-slave-khd3 krb5kdc[24482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440544, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:35:44.550740 24722 init.cc:377] Logged in from keytab as kudu/127.6.33.2@KRBTEST.COM (short username kudu)
I20260926 16:35:44.552836 24722 webserver.cc:460] Webserver started at http://127.6.33.2:44635/ using document root <none> and password file <none>
I20260926 16:35:44.553663 24722 fs_manager.cc:359] Metadata directory not provided
I20260926 16:35:44.553781 24722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:35:44.554139 24722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:35:44.557538 24722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/ts-1/data/instance:
uuid: "91aaf549c66a43b799a51c89756bc40a"
format_stamp: "Formatted at 2026-09-26 16:35:44 on dist-test-slave-khd3"
server_key: "12031d2ecf73c0424723bbfe5518ef53"
server_key_iv: "76e78fd3d69be7242436af874a799748"
server_key_version: "encryptionkey@0"
I20260926 16:35:44.558422 24722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/ts-1/wal/instance:
uuid: "91aaf549c66a43b799a51c89756bc40a"
format_stamp: "Formatted at 2026-09-26 16:35:44 on dist-test-slave-khd3"
server_key: "12031d2ecf73c0424723bbfe5518ef53"
server_key_iv: "76e78fd3d69be7242436af874a799748"
server_key_version: "encryptionkey@0"
I20260926 16:35:44.564107 24722 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 16:35:44.567965 24737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:35:44.569638 24722 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:35:44.569785 24722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/ts-1/wal
uuid: "91aaf549c66a43b799a51c89756bc40a"
format_stamp: "Formatted at 2026-09-26 16:35:44 on dist-test-slave-khd3"
server_key: "12031d2ecf73c0424723bbfe5518ef53"
server_key_iv: "76e78fd3d69be7242436af874a799748"
server_key_version: "encryptionkey@0"
I20260926 16:35:44.569989 24722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:35:44.582877 24722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:35:44.588243 24722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:35:44.588656 24722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:35:44.589708 24722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:35:44.591048 24722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:35:44.591125 24722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:35:44.591213 24722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:35:44.591259 24722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:35:44.626178 24722 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.2:41063
I20260926 16:35:44.626223 24849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.2:41063 every 8 connection(s)
I20260926 16:35:44.627457 24722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
I20260926 16:35:44.632913 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 24722
I20260926 16:35:44.633098 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/ts-1/wal/instance
I20260926 16:35:44.633652 6276 external_mini_cluster.cc:1428] Setting key 38293704e559ea686d0991d47f32c579
Sep 26 16:35:44 dist-test-slave-khd3 krb5kdc[24482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440544, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.2@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:35:44.654400 24852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:44.630126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:36811 (local address 127.6.33.62:34389)
0926 16:35:44.630592 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:44.630597 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:44.631545 (+ 948us) server_negotiation.cc:407] Connection header received
0926 16:35:44.633985 (+ 2440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:44.633989 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:44.634106 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:44.634214 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:35:44.636082 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:44.636980 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:44.639658 (+ 2678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:44.640444 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:44.640598 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:44.648320 (+ 7722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:35:44.648325 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:35:44.648328 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:35:44.648363 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:35:44.651048 (+ 2685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:35:44.651791 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:35:44.651797 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:35:44.651800 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:35:44.651936 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:35:44.652387 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:35:44.652393 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:35:44.652396 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:35:44.652691 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:35:44.652882 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:44.653674 (+ 792us) server_negotiation.cc:299] Negotiation successful
0926 16:35:44.653912 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":263,"threads_started":1}
I20260926 16:35:44.655483 24853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:44.630693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:34389 (local address 127.6.33.2:36811)
0926 16:35:44.631383 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:44.631426 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 16:35:44.633684 (+ 2258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:35:44.634456 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:35:44.634465 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:35:44.635149 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:35:44.635906 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:44.635932 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:44.637159 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:44.637163 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:44.639478 (+ 2315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:44.639502 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:44.640713 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:44.640724 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:44.641116 (+ 392us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:44.645017 (+ 3901us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:35:44.645054 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:35:44.648043 (+ 2989us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:35:44.651205 (+ 3162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:35:44.651216 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:35:44.651235 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:35:44.651658 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:35:44.652069 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:35:44.652074 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:35:44.652078 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:35:44.652270 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:35:44.652859 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:35:44.652867 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:35:44.653290 (+ 423us) client_negotiation.cc:769] Sending connection context
0926 16:35:44.653571 (+ 281us) client_negotiation.cc:240] Negotiation successful
0926 16:35:44.654153 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":270,"threads_started":1}
I20260926 16:35:44.657299 24850 heartbeater.cc:347] Connected to a master server at 127.6.33.62:34389
I20260926 16:35:44.657662 24850 heartbeater.cc:464] Registering TS with master...
I20260926 16:35:44.658457 24850 heartbeater.cc:511] Master 127.6.33.62:34389 requested a full tablet report, sending...
I20260926 16:35:44.660228 24527 ts_manager.cc:194] Registered new tserver with Master: 91aaf549c66a43b799a51c89756bc40a (127.6.33.2:41063)
WARNING: no policy specified for kudu/127.6.33.3@KRBTEST.COM; defaulting to no policy
I20260926 16:35:44.661191 24527 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.33.2@KRBTEST.COM'} at 127.6.33.2:36811
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab.
Entry for principal kudu/127.6.33.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab.
I20260926 16:35:44.694460 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.3:0
--local_ip_for_outbound_sockets=127.6.33.3
--webserver_interface=127.6.33.3
--webserver_port=0
--tserver_master_addrs=127.6.33.62:34389
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab
--principal=kudu/127.6.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:45679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:35:45.034807 24856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:35:45.035133 24856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:35:45.035298 24856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:35:45.043149 24856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:35:45.043282 24856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:35:45.043460 24856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.3
I20260926 16:35:45.055405 24856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:45679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab
--principal=kudu/127.6.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.33.3
--webserver_port=0
--tserver_master_addrs=127.6.33.62:34389
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.3
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:35:45.056954 24856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:35:45.058522 24856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:35:45.072170 24861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:35:45.072124 24862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:35:45.072692 24864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:35:45.073107 24856 server_base.cc:1034] running on GCE node
I20260926 16:35:45.073755 24856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:35:45.074914 24856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:35:45.076181 24856 hybrid_clock.cc:648] HybridClock initialized: now 1790440545076118 us; error 57 us; skew 500 ppm
I20260926 16:35:45.076529 24856 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:35:45 dist-test-slave-khd3 krb5kdc[24482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440545, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:35:45.080168 24856 init.cc:377] Logged in from keytab as kudu/127.6.33.3@KRBTEST.COM (short username kudu)
I20260926 16:35:45.081795 24856 webserver.cc:460] Webserver started at http://127.6.33.3:40287/ using document root <none> and password file <none>
I20260926 16:35:45.082530 24856 fs_manager.cc:359] Metadata directory not provided
I20260926 16:35:45.082679 24856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:35:45.082994 24856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:35:45.086114 24856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/ts-2/data/instance:
uuid: "62acd4046a204d26a033d7433f3a16de"
format_stamp: "Formatted at 2026-09-26 16:35:45 on dist-test-slave-khd3"
server_key: "1901d30e3e10b7bdd0e1db0ddae2fdcc"
server_key_iv: "e6c36553a63244ef34c12b514cd6bdf5"
server_key_version: "encryptionkey@0"
I20260926 16:35:45.086962 24856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/ts-2/wal/instance:
uuid: "62acd4046a204d26a033d7433f3a16de"
format_stamp: "Formatted at 2026-09-26 16:35:45 on dist-test-slave-khd3"
server_key: "1901d30e3e10b7bdd0e1db0ddae2fdcc"
server_key_iv: "e6c36553a63244ef34c12b514cd6bdf5"
server_key_version: "encryptionkey@0"
I20260926 16:35:45.092351 24856 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260926 16:35:45.096182 24871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:35:45.097894 24856 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:35:45.098143 24856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/ts-2/wal
uuid: "62acd4046a204d26a033d7433f3a16de"
format_stamp: "Formatted at 2026-09-26 16:35:45 on dist-test-slave-khd3"
server_key: "1901d30e3e10b7bdd0e1db0ddae2fdcc"
server_key_iv: "e6c36553a63244ef34c12b514cd6bdf5"
server_key_version: "encryptionkey@0"
I20260926 16:35:45.098382 24856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:35:45.118768 24856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:35:45.123402 24856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:35:45.123734 24856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:35:45.124864 24856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:35:45.126305 24856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:35:45.126410 24856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:35:45.126570 24856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:35:45.126650 24856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:35:45.141613 24716 heartbeater.cc:503] Master 127.6.33.62:34389 was elected leader, sending a full tablet report...
I20260926 16:35:45.163666 24856 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.3:39987
I20260926 16:35:45.163702 24983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.3:39987 every 8 connection(s)
I20260926 16:35:45.165126 24856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
I20260926 16:35:45.170285 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 24856
I20260926 16:35:45.170481 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/ts-2/wal/instance
I20260926 16:35:45.170902 6276 external_mini_cluster.cc:1428] Setting key 332bf924143a9d97facbf127f0c8d7e6
Sep 26 16:35:45 dist-test-slave-khd3 krb5kdc[24482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440545, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.3@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:35:45.187443 24986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:45.168148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.3:49709 (local address 127.6.33.62:34389)
0926 16:35:45.168699 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:45.168704 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:45.169716 (+ 1012us) server_negotiation.cc:407] Connection header received
0926 16:35:45.171922 (+ 2206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:45.171928 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:45.172028 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:45.172151 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:35:45.173919 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:45.174904 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:45.176589 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:45.177104 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:45.177246 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:45.180936 (+ 3690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:35:45.180942 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:35:45.180945 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:35:45.180980 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:35:45.183856 (+ 2876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:35:45.184720 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:35:45.184725 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:35:45.184729 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:35:45.184918 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:35:45.185463 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:35:45.185468 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:35:45.185472 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:35:45.185818 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:35:45.186005 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:45.186774 (+ 769us) server_negotiation.cc:299] Negotiation successful
0926 16:35:45.187027 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":317,"threads_started":1}
I20260926 16:35:45.188454 24987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:45.168692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:34389 (local address 127.6.33.3:49709)
0926 16:35:45.169546 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:45.169584 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 16:35:45.171626 (+ 2042us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:35:45.172399 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:35:45.172408 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:35:45.173055 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:35:45.173754 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:45.173775 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:45.175064 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:45.175068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:45.176431 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:45.176443 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:45.177347 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:45.177358 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:45.177627 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:45.178303 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:35:45.178332 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:35:45.180672 (+ 2340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:35:45.184050 (+ 3378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:35:45.184060 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:35:45.184086 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:35:45.184579 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:35:45.185110 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:35:45.185115 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:35:45.185119 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:35:45.185329 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:35:45.185950 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:35:45.185958 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:35:45.186383 (+ 425us) client_negotiation.cc:769] Sending connection context
0926 16:35:45.186674 (+ 291us) client_negotiation.cc:240] Negotiation successful
0926 16:35:45.187272 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":318,"threads_started":1}
I20260926 16:35:45.189968 24984 heartbeater.cc:347] Connected to a master server at 127.6.33.62:34389
I20260926 16:35:45.190327 24984 heartbeater.cc:464] Registering TS with master...
I20260926 16:35:45.191076 24984 heartbeater.cc:511] Master 127.6.33.62:34389 requested a full tablet report, sending...
I20260926 16:35:45.192596 24527 ts_manager.cc:194] Registered new tserver with Master: 62acd4046a204d26a033d7433f3a16de (127.6.33.3:39987)
I20260926 16:35:45.193533 24527 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.33.3@KRBTEST.COM'} at 127.6.33.3:49709
I20260926 16:35:45.199081 6276 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:35:45.217253 24986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:45.203380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46186 (local address 127.6.33.62:34389)
0926 16:35:45.203639 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:45.203648 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:45.204044 (+ 396us) server_negotiation.cc:407] Connection header received
0926 16:35:45.204268 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:45.204273 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:45.204352 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:45.204460 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:35:45.206018 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:45.206780 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:45.208110 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:45.208548 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:45.208693 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:45.211760 (+ 3067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:35:45.211771 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:35:45.211775 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:35:45.211808 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:35:45.214340 (+ 2532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:35:45.215130 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:35:45.215142 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:35:45.215145 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:35:45.215266 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:35:45.215733 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:35:45.215740 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:35:45.215747 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:35:45.216061 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:35:45.216339 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:45.216713 (+ 374us) server_negotiation.cc:299] Negotiation successful
0926 16:35:45.216966 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 16:35:45.221771 24527 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46186:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:35:45.228878 24527 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:35:45.255658 24997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:45.241520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:35595 (local address 127.0.0.1:49520)
0926 16:35:45.242417 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:45.242475 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 16:35:45.242710 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:35:45.245017 (+ 2307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:35:45.245025 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:35:45.245062 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:35:45.245777 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:45.245795 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:45.248733 (+ 2938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:45.248747 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:45.251880 (+ 3133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:45.251903 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:45.254592 (+ 2689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:45.254600 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:45.254786 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:45.254821 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 16:35:45.254893 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 16:35:45.255089 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":463,"threads_started":1}
I20260926 16:35:45.256191 24999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:45.241764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49520 (local address 127.6.33.1:35595)
0926 16:35:45.244208 (+ 2444us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:45.244221 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:45.244260 (+ 39us) server_negotiation.cc:407] Connection header received
0926 16:35:45.244395 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:45.244406 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:45.244696 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:45.246072 (+ 1376us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:35:45.246381 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:45.248529 (+ 2148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:45.252168 (+ 3639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:45.254328 (+ 2160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:45.255242 (+ 914us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:45.255573 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:45.255653 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 16:35:45.255833 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2219,"thread_start_us":2133,"threads_started":1}
I20260926 16:35:45.256454 24998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:45.241673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.2:41063 (local address 127.0.0.1:40746)
0926 16:35:45.243412 (+ 1739us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:45.243438 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 16:35:45.243572 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:35:45.244699 (+ 1127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:35:45.244708 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:35:45.244742 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:35:45.245388 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:45.245409 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:45.247979 (+ 2570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:45.247984 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:45.251513 (+ 3529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:45.251537 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:45.254234 (+ 2697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:45.254244 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:45.254578 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:45.254646 (+ 68us) client_negotiation.cc:769] Sending connection context
0926 16:35:45.254763 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 16:35:45.254969 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1549,"thread_start_us":1312,"threads_started":1}
I20260926 16:35:45.256997 25000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:45.242106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40746 (local address 127.6.33.2:41063)
0926 16:35:45.243941 (+ 1835us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:45.243950 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:45.243982 (+ 32us) server_negotiation.cc:407] Connection header received
0926 16:35:45.244110 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:45.244119 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:45.244419 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:45.245694 (+ 1275us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:35:45.245993 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:45.247810 (+ 1817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:45.251733 (+ 3923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:45.253941 (+ 2208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:45.254878 (+ 937us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:45.255104 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:45.255187 (+ 83us) server_negotiation.cc:299] Negotiation successful
0926 16:35:45.255361 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1660,"thread_start_us":2244,"threads_started":1}
I20260926 16:35:45.260512 24651 tablet_service.cc:1467] Processing CreateTablet for tablet 9c54babc91b14c8cbdb72272daa238b5 (DEFAULT_TABLE table=test-table [id=06df08aecaa94e5996adc23791aef795]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:35:45.260542 24785 tablet_service.cc:1467] Processing CreateTablet for tablet 9c54babc91b14c8cbdb72272daa238b5 (DEFAULT_TABLE table=test-table [id=06df08aecaa94e5996adc23791aef795]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:35:45.261998 24651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c54babc91b14c8cbdb72272daa238b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:35:45.262006 24785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c54babc91b14c8cbdb72272daa238b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:35:45.264721 24998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:45.243754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.3:39987 (local address 127.0.0.1:37070)
0926 16:35:45.256716 (+ 12962us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:45.256736 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 16:35:45.256867 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:35:45.257717 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:35:45.257722 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:35:45.257748 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:35:45.257915 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:45.257926 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:45.260170 (+ 2244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:45.260176 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:45.262105 (+ 1929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:45.262122 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:45.263919 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:45.263926 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:45.264098 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:45.264147 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 16:35:45.264267 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 16:35:45.264431 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12822,"thread_start_us":13768,"threads_started":1}
I20260926 16:35:45.264968 25002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:45.244208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37070 (local address 127.6.33.3:39987)
0926 16:35:45.254896 (+ 10688us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:45.254907 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:45.257047 (+ 2140us) server_negotiation.cc:407] Connection header received
0926 16:35:45.257186 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:45.257194 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:45.257479 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:45.258106 (+ 627us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:35:45.258405 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:45.259997 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:45.262305 (+ 2308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:45.263733 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:45.263888 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:45.264110 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:45.264344 (+ 234us) server_negotiation.cc:299] Negotiation successful
0926 16:35:45.264509 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10498,"thread_start_us":10333,"threads_started":1}
I20260926 16:35:45.269510 24919 tablet_service.cc:1467] Processing CreateTablet for tablet 9c54babc91b14c8cbdb72272daa238b5 (DEFAULT_TABLE table=test-table [id=06df08aecaa94e5996adc23791aef795]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:35:45.271100 24919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c54babc91b14c8cbdb72272daa238b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:35:45.275895 25003 tablet_bootstrap.cc:492] T 9c54babc91b14c8cbdb72272daa238b5 P 91aaf549c66a43b799a51c89756bc40a: Bootstrap starting.
I20260926 16:35:45.278101 25004 tablet_bootstrap.cc:492] T 9c54babc91b14c8cbdb72272daa238b5 P f9ae241ae3694966bb900e5546bc2fb9: Bootstrap starting.
I20260926 16:35:45.279732 25003 tablet_bootstrap.cc:654] T 9c54babc91b14c8cbdb72272daa238b5 P 91aaf549c66a43b799a51c89756bc40a: Neither blocks nor log segments found. Creating new log.
I20260926 16:35:45.281126 25003 log.cc:824] T 9c54babc91b14c8cbdb72272daa238b5 P 91aaf549c66a43b799a51c89756bc40a: Log is configured to *not* fsync() on all Append() calls
I20260926 16:35:45.281908 25004 tablet_bootstrap.cc:654] T 9c54babc91b14c8cbdb72272daa238b5 P f9ae241ae3694966bb900e5546bc2fb9: Neither blocks nor log segments found. Creating new log.
I20260926 16:35:45.283108 25004 log.cc:824] T 9c54babc91b14c8cbdb72272daa238b5 P f9ae241ae3694966bb900e5546bc2fb9: Log is configured to *not* fsync() on all Append() calls
I20260926 16:35:45.284313 25003 tablet_bootstrap.cc:492] T 9c54babc91b14c8cbdb72272daa238b5 P 91aaf549c66a43b799a51c89756bc40a: No bootstrap required, opened a new log
I20260926 16:35:45.284633 25003 ts_tablet_manager.cc:1397] T 9c54babc91b14c8cbdb72272daa238b5 P 91aaf549c66a43b799a51c89756bc40a: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260926 16:35:45.286352 25004 tablet_bootstrap.cc:492] T 9c54babc91b14c8cbdb72272daa238b5 P f9ae241ae3694966bb900e5546bc2fb9: No bootstrap required, opened a new log
I20260926 16:35:45.286707 25004 ts_tablet_manager.cc:1397] T 9c54babc91b14c8cbdb72272daa238b5 P f9ae241ae3694966bb900e5546bc2fb9: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.002s
I20260926 16:35:45.288064 25007 tablet_bootstrap.cc:492] T 9c54babc91b14c8cbdb72272daa238b5 P 62acd4046a204d26a033d7433f3a16de: Bootstrap starting.
I20260926 16:35:45.291118 25007 tablet_bootstrap.cc:654] T 9c54babc91b14c8cbdb72272daa238b5 P 62acd4046a204d26a033d7433f3a16de: Neither blocks nor log segments found. Creating new log.
I20260926 16:35:45.292219 25007 log.cc:824] T 9c54babc91b14c8cbdb72272daa238b5 P 62acd4046a204d26a033d7433f3a16de: Log is configured to *not* fsync() on all Append() calls
I20260926 16:35:45.295122 25007 tablet_bootstrap.cc:492] T 9c54babc91b14c8cbdb72272daa238b5 P 62acd4046a204d26a033d7433f3a16de: No bootstrap required, opened a new log
I20260926 16:35:45.295424 25007 ts_tablet_manager.cc:1397] T 9c54babc91b14c8cbdb72272daa238b5 P 62acd4046a204d26a033d7433f3a16de: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260926 16:35:45.295699 25003 raft_consensus.cc:348] T 9c54babc91b14c8cbdb72272daa238b5 P 91aaf549c66a43b799a51c89756bc40a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9ae241ae3694966bb900e5546bc2fb9" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 35595 } } peers { permanent_uuid: "91aaf549c66a43b799a51c89756bc40a" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 41063 } } peers { permanent_uuid: "62acd4046a204d26a033d7433f3a16de" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39987 } }
I20260926 16:35:45.296181 25003 raft_consensus.cc:374] T 9c54babc91b14c8cbdb72272daa238b5 P 91aaf549c66a43b799a51c89756bc40a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:35:45.296280 25003 raft_consensus.cc:729] T 9c54babc91b14c8cbdb72272daa238b5 P 91aaf549c66a43b799a51c89756bc40a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91aaf549c66a43b799a51c89756bc40a, State: Initialized, Role: FOLLOWER
I20260926 16:35:45.297103 25003 consensus_queue.cc:260] T 9c54babc91b14c8cbdb72272daa238b5 P 91aaf549c66a43b799a51c89756bc40a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9ae241ae3694966bb900e5546bc2fb9" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 35595 } } peers { permanent_uuid: "91aaf549c66a43b799a51c89756bc40a" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 41063 } } peers { permanent_uuid: "62acd4046a204d26a033d7433f3a16de" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39987 } }
I20260926 16:35:45.297736 25004 raft_consensus.cc:348] T 9c54babc91b14c8cbdb72272daa238b5 P f9ae241ae3694966bb900e5546bc2fb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9ae241ae3694966bb900e5546bc2fb9" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 35595 } } peers { permanent_uuid: "91aaf549c66a43b799a51c89756bc40a" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 41063 } } peers { permanent_uuid: "62acd4046a204d26a033d7433f3a16de" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39987 } }
I20260926 16:35:45.298321 25004 raft_consensus.cc:374] T 9c54babc91b14c8cbdb72272daa238b5 P f9ae241ae3694966bb900e5546bc2fb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:35:45.298432 25004 raft_consensus.cc:729] T 9c54babc91b14c8cbdb72272daa238b5 P f9ae241ae3694966bb900e5546bc2fb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9ae241ae3694966bb900e5546bc2fb9, State: Initialized, Role: FOLLOWER
I20260926 16:35:45.299220 24850 heartbeater.cc:503] Master 127.6.33.62:34389 was elected leader, sending a full tablet report...
I20260926 16:35:45.299104 25004 consensus_queue.cc:260] T 9c54babc91b14c8cbdb72272daa238b5 P f9ae241ae3694966bb900e5546bc2fb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9ae241ae3694966bb900e5546bc2fb9" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 35595 } } peers { permanent_uuid: "91aaf549c66a43b799a51c89756bc40a" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 41063 } } peers { permanent_uuid: "62acd4046a204d26a033d7433f3a16de" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39987 } }
I20260926 16:35:45.300019 25003 ts_tablet_manager.cc:1428] T 9c54babc91b14c8cbdb72272daa238b5 P 91aaf549c66a43b799a51c89756bc40a: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260926 16:35:45.301017 25004 ts_tablet_manager.cc:1428] T 9c54babc91b14c8cbdb72272daa238b5 P f9ae241ae3694966bb900e5546bc2fb9: Time spent starting tablet: real 0.014s user 0.012s sys 0.002s
I20260926 16:35:45.307101 25007 raft_consensus.cc:348] T 9c54babc91b14c8cbdb72272daa238b5 P 62acd4046a204d26a033d7433f3a16de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9ae241ae3694966bb900e5546bc2fb9" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 35595 } } peers { permanent_uuid: "91aaf549c66a43b799a51c89756bc40a" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 41063 } } peers { permanent_uuid: "62acd4046a204d26a033d7433f3a16de" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39987 } }
I20260926 16:35:45.307508 25007 raft_consensus.cc:374] T 9c54babc91b14c8cbdb72272daa238b5 P 62acd4046a204d26a033d7433f3a16de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:35:45.307632 25007 raft_consensus.cc:729] T 9c54babc91b14c8cbdb72272daa238b5 P 62acd4046a204d26a033d7433f3a16de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62acd4046a204d26a033d7433f3a16de, State: Initialized, Role: FOLLOWER
I20260926 16:35:45.308262 25007 consensus_queue.cc:260] T 9c54babc91b14c8cbdb72272daa238b5 P 62acd4046a204d26a033d7433f3a16de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9ae241ae3694966bb900e5546bc2fb9" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 35595 } } peers { permanent_uuid: "91aaf549c66a43b799a51c89756bc40a" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 41063 } } peers { permanent_uuid: "62acd4046a204d26a033d7433f3a16de" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39987 } }
I20260926 16:35:45.309362 24984 heartbeater.cc:503] Master 127.6.33.62:34389 was elected leader, sending a full tablet report...
I20260926 16:35:45.310119 25007 ts_tablet_manager.cc:1428] T 9c54babc91b14c8cbdb72272daa238b5 P 62acd4046a204d26a033d7433f3a16de: Time spent starting tablet: real 0.015s user 0.007s sys 0.006s
W20260926 16:35:45.358960 24717 tablet.cc:2369] T 9c54babc91b14c8cbdb72272daa238b5 P f9ae241ae3694966bb900e5546bc2fb9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 16:35:45.379477 24851 tablet.cc:2369] T 9c54babc91b14c8cbdb72272daa238b5 P 91aaf549c66a43b799a51c89756bc40a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 16:35:45.417099 24985 tablet.cc:2369] T 9c54babc91b14c8cbdb72272daa238b5 P 62acd4046a204d26a033d7433f3a16de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:35:45.579633 25009 raft_consensus.cc:482] T 9c54babc91b14c8cbdb72272daa238b5 P 91aaf549c66a43b799a51c89756bc40a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:35:45.579960 25009 raft_consensus.cc:504] T 9c54babc91b14c8cbdb72272daa238b5 P 91aaf549c66a43b799a51c89756bc40a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9ae241ae3694966bb900e5546bc2fb9" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 35595 } } peers { permanent_uuid: "91aaf549c66a43b799a51c89756bc40a" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 41063 } } peers { permanent_uuid: "62acd4046a204d26a033d7433f3a16de" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39987 } }
I20260926 16:35:45.581915 25009 leader_election.cc:288] T 9c54babc91b14c8cbdb72272daa238b5 P 91aaf549c66a43b799a51c89756bc40a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9ae241ae3694966bb900e5546bc2fb9 (127.6.33.1:35595), 62acd4046a204d26a033d7433f3a16de (127.6.33.3:39987)
I20260926 16:35:45.588935 24999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:45.582409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:48689 (local address 127.6.33.1:35595)
0926 16:35:45.582656 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:45.582665 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:45.583368 (+ 703us) server_negotiation.cc:407] Connection header received
0926 16:35:45.583598 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:45.583603 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:45.583662 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:45.583764 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:35:45.584333 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:45.585124 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:45.587065 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:45.587961 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:45.588109 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:45.588223 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:45.588514 (+ 291us) server_negotiation.cc:299] Negotiation successful
0926 16:35:45.588650 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 16:35:45.589037 25002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:45.582354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:51919 (local address 127.6.33.3:39987)
0926 16:35:45.582578 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:45.582587 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:45.583523 (+ 936us) server_negotiation.cc:407] Connection header received
0926 16:35:45.583728 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:45.583733 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:45.583791 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:45.583895 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:35:45.584499 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:45.585299 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:45.587250 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:45.588078 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:45.588224 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:45.588337 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:45.588629 (+ 292us) server_negotiation.cc:299] Negotiation successful
0926 16:35:45.588777 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 16:35:45.589057 25013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:45.582144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:35595 (local address 127.6.33.2:48689)
0926 16:35:45.583247 (+ 1103us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:45.583263 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:35:45.583442 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:35:45.583780 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:35:45.583785 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:35:45.583808 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:35:45.584184 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:45.584196 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:45.585253 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:45.585260 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:45.586911 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:45.586923 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:45.588191 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:45.588197 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:45.588352 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:45.588379 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 16:35:45.588490 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 16:35:45.588621 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":968,"mutex_wait_us":1,"thread_start_us":973,"threads_started":1}
I20260926 16:35:45.589545 25012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:45.582337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.3:39987 (local address 127.6.33.2:51919)
0926 16:35:45.583410 (+ 1073us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:45.583424 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 16:35:45.583572 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:35:45.583913 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:35:45.583917 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:35:45.583955 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:35:45.584351 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:45.584360 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:45.585442 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:45.585448 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:45.587040 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:45.587052 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:45.588296 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:45.588302 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:45.588463 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:45.588484 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 16:35:45.588585 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 16:35:45.588708 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":970,"mutex_wait_us":74,"thread_start_us":425,"threads_started":1}
I20260926 16:35:45.590195 24671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c54babc91b14c8cbdb72272daa238b5" candidate_uuid: "91aaf549c66a43b799a51c89756bc40a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9ae241ae3694966bb900e5546bc2fb9" is_pre_election: true
I20260926 16:35:45.590541 24939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c54babc91b14c8cbdb72272daa238b5" candidate_uuid: "91aaf549c66a43b799a51c89756bc40a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62acd4046a204d26a033d7433f3a16de" is_pre_election: true
I20260926 16:35:45.590821 24671 raft_consensus.cc:2454] T 9c54babc91b14c8cbdb72272daa238b5 P f9ae241ae3694966bb900e5546bc2fb9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91aaf549c66a43b799a51c89756bc40a in term 0.
I20260926 16:35:45.591070 24939 raft_consensus.cc:2454] T 9c54babc91b14c8cbdb72272daa238b5 P 62acd4046a204d26a033d7433f3a16de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91aaf549c66a43b799a51c89756bc40a in term 0.
I20260926 16:35:45.591670 24741 leader_election.cc:302] T 9c54babc91b14c8cbdb72272daa238b5 P 91aaf549c66a43b799a51c89756bc40a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91aaf549c66a43b799a51c89756bc40a, f9ae241ae3694966bb900e5546bc2fb9; no voters:
I20260926 16:35:45.592083 25009 raft_consensus.cc:2781] T 9c54babc91b14c8cbdb72272daa238b5 P 91aaf549c66a43b799a51c89756bc40a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:35:45.592231 25009 raft_consensus.cc:482] T 9c54babc91b14c8cbdb72272daa238b5 P 91aaf549c66a43b799a51c89756bc40a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:35:45.592367 25009 raft_consensus.cc:3037] T 9c54babc91b14c8cbdb72272daa238b5 P 91aaf549c66a43b799a51c89756bc40a [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:35:45.594743 25009 raft_consensus.cc:504] T 9c54babc91b14c8cbdb72272daa238b5 P 91aaf549c66a43b799a51c89756bc40a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9ae241ae3694966bb900e5546bc2fb9" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 35595 } } peers { permanent_uuid: "91aaf549c66a43b799a51c89756bc40a" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 41063 } } peers { permanent_uuid: "62acd4046a204d26a033d7433f3a16de" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39987 } }
I20260926 16:35:45.595356 25009 leader_election.cc:288] T 9c54babc91b14c8cbdb72272daa238b5 P 91aaf549c66a43b799a51c89756bc40a [CANDIDATE]: Term 1 election: Requested vote from peers f9ae241ae3694966bb900e5546bc2fb9 (127.6.33.1:35595), 62acd4046a204d26a033d7433f3a16de (127.6.33.3:39987)
I20260926 16:35:45.595839 24671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c54babc91b14c8cbdb72272daa238b5" candidate_uuid: "91aaf549c66a43b799a51c89756bc40a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9ae241ae3694966bb900e5546bc2fb9"
I20260926 16:35:45.595885 24939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c54babc91b14c8cbdb72272daa238b5" candidate_uuid: "91aaf549c66a43b799a51c89756bc40a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62acd4046a204d26a033d7433f3a16de"
I20260926 16:35:45.596045 24671 raft_consensus.cc:3037] T 9c54babc91b14c8cbdb72272daa238b5 P f9ae241ae3694966bb900e5546bc2fb9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:35:45.596153 24939 raft_consensus.cc:3037] T 9c54babc91b14c8cbdb72272daa238b5 P 62acd4046a204d26a033d7433f3a16de [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:35:45.598410 24939 raft_consensus.cc:2454] T 9c54babc91b14c8cbdb72272daa238b5 P 62acd4046a204d26a033d7433f3a16de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91aaf549c66a43b799a51c89756bc40a in term 1.
I20260926 16:35:45.598418 24671 raft_consensus.cc:2454] T 9c54babc91b14c8cbdb72272daa238b5 P f9ae241ae3694966bb900e5546bc2fb9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91aaf549c66a43b799a51c89756bc40a in term 1.
I20260926 16:35:45.598868 24740 leader_election.cc:302] T 9c54babc91b14c8cbdb72272daa238b5 P 91aaf549c66a43b799a51c89756bc40a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62acd4046a204d26a033d7433f3a16de, 91aaf549c66a43b799a51c89756bc40a; no voters:
I20260926 16:35:45.599190 25009 raft_consensus.cc:2781] T 9c54babc91b14c8cbdb72272daa238b5 P 91aaf549c66a43b799a51c89756bc40a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:35:45.599641 25009 raft_consensus.cc:686] T 9c54babc91b14c8cbdb72272daa238b5 P 91aaf549c66a43b799a51c89756bc40a [term 1 LEADER]: Becoming Leader. State: Replica: 91aaf549c66a43b799a51c89756bc40a, State: Running, Role: LEADER
I20260926 16:35:45.600145 25009 consensus_queue.cc:237] T 9c54babc91b14c8cbdb72272daa238b5 P 91aaf549c66a43b799a51c89756bc40a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9ae241ae3694966bb900e5546bc2fb9" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 35595 } } peers { permanent_uuid: "91aaf549c66a43b799a51c89756bc40a" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 41063 } } peers { permanent_uuid: "62acd4046a204d26a033d7433f3a16de" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39987 } }
I20260926 16:35:45.604943 24527 catalog_manager.cc:5495] T 9c54babc91b14c8cbdb72272daa238b5 P 91aaf549c66a43b799a51c89756bc40a reported cstate change: term changed from 0 to 1, leader changed from <none> to 91aaf549c66a43b799a51c89756bc40a (127.6.33.2). New cstate: current_term: 1 leader_uuid: "91aaf549c66a43b799a51c89756bc40a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9ae241ae3694966bb900e5546bc2fb9" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 35595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91aaf549c66a43b799a51c89756bc40a" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 41063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "62acd4046a204d26a033d7433f3a16de" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39987 } health_report { overall_health: UNKNOWN } } }
I20260926 16:35:45.629721 25000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:45.621518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40754 (local address 127.6.33.2:41063)
0926 16:35:45.621700 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:45.621704 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:45.621740 (+ 36us) server_negotiation.cc:407] Connection header received
0926 16:35:45.622028 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:45.622034 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:45.622156 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:45.622331 (+ 175us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:35:45.622946 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:45.624198 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:45.626498 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:45.627175 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:45.627310 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:45.628360 (+ 1050us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:35:45.629080 (+ 720us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:35:45.629211 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:45.629451 (+ 240us) server_negotiation.cc:299] Negotiation successful
0926 16:35:45.629497 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 16:35:45.639190 24671 raft_consensus.cc:1261] T 9c54babc91b14c8cbdb72272daa238b5 P f9ae241ae3694966bb900e5546bc2fb9 [term 1 FOLLOWER]: Refusing update from remote peer 91aaf549c66a43b799a51c89756bc40a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:35:45.639253 24939 raft_consensus.cc:1261] T 9c54babc91b14c8cbdb72272daa238b5 P 62acd4046a204d26a033d7433f3a16de [term 1 FOLLOWER]: Refusing update from remote peer 91aaf549c66a43b799a51c89756bc40a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:35:45.640200 25009 consensus_queue.cc:1035] T 9c54babc91b14c8cbdb72272daa238b5 P 91aaf549c66a43b799a51c89756bc40a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9ae241ae3694966bb900e5546bc2fb9" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:35:45.640542 25014 consensus_queue.cc:1035] T 9c54babc91b14c8cbdb72272daa238b5 P 91aaf549c66a43b799a51c89756bc40a [LEADER]: Connected to new peer: Peer: permanent_uuid: "62acd4046a204d26a033d7433f3a16de" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:35:45.667879 24526 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46186:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:35:45.668227 24526 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46186:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:35:45.673010 24526 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2e2480dddece4a42a57d58e5f3928b8b: Sending DeleteTablet for 3 replicas of tablet 9c54babc91b14c8cbdb72272daa238b5
I20260926 16:35:45.674361 24651 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c54babc91b14c8cbdb72272daa238b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:35:45 UTC) from {username='kudu', principal='kudu/127.6.33.62@KRBTEST.COM'} at 127.0.0.1:49520
I20260926 16:35:45.674378 24785 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c54babc91b14c8cbdb72272daa238b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:35:45 UTC) from {username='kudu', principal='kudu/127.6.33.62@KRBTEST.COM'} at 127.0.0.1:40746
I20260926 16:35:45.674396 24919 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c54babc91b14c8cbdb72272daa238b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:35:45 UTC) from {username='kudu', principal='kudu/127.6.33.62@KRBTEST.COM'} at 127.0.0.1:37070
I20260926 16:35:45.675639 25029 tablet_replica.cc:330] T 9c54babc91b14c8cbdb72272daa238b5 P 91aaf549c66a43b799a51c89756bc40a: stopping tablet replica
I20260926 16:35:45.676242 25029 raft_consensus.cc:2229] T 9c54babc91b14c8cbdb72272daa238b5 P 91aaf549c66a43b799a51c89756bc40a [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:35:45.677320 25029 raft_consensus.cc:2258] T 9c54babc91b14c8cbdb72272daa238b5 P 91aaf549c66a43b799a51c89756bc40a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:35:45.678277 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 24588
I20260926 16:35:45.678368 25028 tablet_replica.cc:330] T 9c54babc91b14c8cbdb72272daa238b5 P f9ae241ae3694966bb900e5546bc2fb9: stopping tablet replica
I20260926 16:35:45.678655 25027 tablet_replica.cc:330] T 9c54babc91b14c8cbdb72272daa238b5 P 62acd4046a204d26a033d7433f3a16de: stopping tablet replica
I20260926 16:35:45.678853 25028 raft_consensus.cc:2229] T 9c54babc91b14c8cbdb72272daa238b5 P f9ae241ae3694966bb900e5546bc2fb9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:35:45.679194 25027 raft_consensus.cc:2229] T 9c54babc91b14c8cbdb72272daa238b5 P 62acd4046a204d26a033d7433f3a16de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:35:45.679451 25028 raft_consensus.cc:2258] T 9c54babc91b14c8cbdb72272daa238b5 P f9ae241ae3694966bb900e5546bc2fb9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:35:45.679838 25027 raft_consensus.cc:2258] T 9c54babc91b14c8cbdb72272daa238b5 P 62acd4046a204d26a033d7433f3a16de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:35:45.682081 25029 ts_tablet_manager.cc:1905] T 9c54babc91b14c8cbdb72272daa238b5 P 91aaf549c66a43b799a51c89756bc40a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:35:45.687883 25028 ts_tablet_manager.cc:1905] T 9c54babc91b14c8cbdb72272daa238b5 P f9ae241ae3694966bb900e5546bc2fb9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:35:45.690672 25029 ts_tablet_manager.cc:1918] T 9c54babc91b14c8cbdb72272daa238b5 P 91aaf549c66a43b799a51c89756bc40a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:35:45.690907 25029 log.cc:1196] T 9c54babc91b14c8cbdb72272daa238b5 P 91aaf549c66a43b799a51c89756bc40a: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/ts-1/wal/wals/9c54babc91b14c8cbdb72272daa238b5
I20260926 16:35:45.691541 25029 ts_tablet_manager.cc:1939] T 9c54babc91b14c8cbdb72272daa238b5 P 91aaf549c66a43b799a51c89756bc40a: Deleting consensus metadata
I20260926 16:35:45.694281 24512 catalog_manager.cc:4841] TS 91aaf549c66a43b799a51c89756bc40a (127.6.33.2:41063): tablet 9c54babc91b14c8cbdb72272daa238b5 (table test-table [id=06df08aecaa94e5996adc23791aef795]) successfully deleted
I20260926 16:35:45.694967 25027 ts_tablet_manager.cc:1905] T 9c54babc91b14c8cbdb72272daa238b5 P 62acd4046a204d26a033d7433f3a16de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:35:45.696578 25028 ts_tablet_manager.cc:1918] T 9c54babc91b14c8cbdb72272daa238b5 P f9ae241ae3694966bb900e5546bc2fb9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:35:45.696787 25028 log.cc:1196] T 9c54babc91b14c8cbdb72272daa238b5 P f9ae241ae3694966bb900e5546bc2fb9: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/ts-0/wal/wals/9c54babc91b14c8cbdb72272daa238b5
I20260926 16:35:45.697388 25028 ts_tablet_manager.cc:1939] T 9c54babc91b14c8cbdb72272daa238b5 P f9ae241ae3694966bb900e5546bc2fb9: Deleting consensus metadata
I20260926 16:35:45.699421 24514 catalog_manager.cc:4841] TS f9ae241ae3694966bb900e5546bc2fb9 (127.6.33.1:35595): tablet 9c54babc91b14c8cbdb72272daa238b5 (table test-table [id=06df08aecaa94e5996adc23791aef795]) successfully deleted
Sep 26 16:35:45 dist-test-slave-khd3 krb5kdc[24482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.1@KRBTEST.COM
I20260926 16:35:45.701910 25027 ts_tablet_manager.cc:1918] T 9c54babc91b14c8cbdb72272daa238b5 P 62acd4046a204d26a033d7433f3a16de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:35:45.702082 25027 log.cc:1196] T 9c54babc91b14c8cbdb72272daa238b5 P 62acd4046a204d26a033d7433f3a16de: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440233850748-6276-0/minicluster-data/ts-2/wal/wals/9c54babc91b14c8cbdb72272daa238b5
I20260926 16:35:45.702670 25027 ts_tablet_manager.cc:1939] T 9c54babc91b14c8cbdb72272daa238b5 P 62acd4046a204d26a033d7433f3a16de: Deleting consensus metadata
I20260926 16:35:45.704660 24513 catalog_manager.cc:4841] TS 62acd4046a204d26a033d7433f3a16de (127.6.33.3:39987): tablet 9c54babc91b14c8cbdb72272daa238b5 (table test-table [id=06df08aecaa94e5996adc23791aef795]) successfully deleted
I20260926 16:35:45.709254 24999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:45.679865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49524 (local address 127.6.33.1:35595)
0926 16:35:45.680075 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:45.680081 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:45.685364 (+ 5283us) server_negotiation.cc:407] Connection header received
0926 16:35:45.685715 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:45.685721 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:45.685838 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:45.685918 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:35:45.688115 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:45.689300 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:45.691614 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:45.692288 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:45.692419 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:45.701717 (+ 9298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:35:45.701736 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:35:45.701759 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:35:45.701816 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:35:45.705962 (+ 4146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:35:45.706788 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:35:45.706798 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:35:45.706802 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:35:45.706969 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:35:45.707401 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:35:45.707407 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:35:45.707411 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:35:45.707907 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:35:45.708228 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:45.708520 (+ 292us) server_negotiation.cc:299] Negotiation successful
0926 16:35:45.708832 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":76}
I20260926 16:35:45.710021 24711 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:35:45.801507 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 24588
I20260926 16:35:45.824277 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 24722
Sep 26 16:35:45 dist-test-slave-khd3 krb5kdc[24482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.2@KRBTEST.COM
I20260926 16:35:45.840821 25000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:45.825126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40766 (local address 127.6.33.2:41063)
0926 16:35:45.825336 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:45.825346 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:45.825543 (+ 197us) server_negotiation.cc:407] Connection header received
0926 16:35:45.825758 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:45.825763 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:45.825853 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:45.825963 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:35:45.827254 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:45.828061 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:45.829573 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:45.830011 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:45.830172 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:45.834531 (+ 4359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:35:45.834546 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:35:45.834561 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:35:45.834601 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:35:45.837793 (+ 3192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:35:45.838535 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:35:45.838542 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:35:45.838545 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:35:45.838699 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:35:45.839081 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:35:45.839087 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:35:45.839090 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:35:45.839451 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:35:45.839803 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:45.840185 (+ 382us) server_negotiation.cc:299] Negotiation successful
0926 16:35:45.840414 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 16:35:45.841593 24845 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:35:45.957844 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 24722
I20260926 16:35:45.976259 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 24856
Sep 26 16:35:45 dist-test-slave-khd3 krb5kdc[24482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.3@KRBTEST.COM
I20260926 16:35:46.002213 25002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:45.977074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37086 (local address 127.6.33.3:39987)
0926 16:35:45.977270 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:45.977275 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:45.977295 (+ 20us) server_negotiation.cc:407] Connection header received
0926 16:35:45.977508 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:45.977513 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:45.977595 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:45.977686 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:35:45.979025 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:45.979857 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:45.981399 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:45.981787 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:45.981893 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:45.995112 (+ 13219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:35:45.995132 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:35:45.995148 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:35:45.995192 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:35:45.998625 (+ 3433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:35:45.999609 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:35:45.999639 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:35:45.999643 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:35:45.999852 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:35:46.000437 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:35:46.000443 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:35:46.000446 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:35:46.001007 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:35:46.001294 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:46.001522 (+ 228us) server_negotiation.cc:299] Negotiation successful
0926 16:35:46.001689 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 16:35:46.002939 24979 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:35:46.093957 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 24856
I20260926 16:35:46.112601 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 24495
I20260926 16:35:46.113816 24557 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:35:46.226516 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 24495
2026-09-26T16:35:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4624 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-26T16:35:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:35:46Z Disabled control of system clock
I20260926 16:35:46.305882 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:44035
--webserver_interface=127.6.33.62
--webserver_port=0
--builtin_ntp_servers=127.6.33.20:35547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.33.62:44035
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 16:35:46.648111 25038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:35:46.648499 25038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:35:46.648593 25038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:35:46.656046 25038 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:35:46.656217 25038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:35:46.656256 25038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:35:46.656308 25038 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:35:46.656352 25038 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:35:46.666225 25038 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:35547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.33.62:44035
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:44035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:35:46.667532 25038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:35:46.669457 25038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:35:46.681854 25043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:35:46.681751 25044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:35:46.681792 25046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:35:46.682991 25038 server_base.cc:1034] running on GCE node
I20260926 16:35:46.683887 25038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:35:46.685461 25038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:35:46.686697 25038 hybrid_clock.cc:648] HybridClock initialized: now 1790440546686653 us; error 38 us; skew 500 ppm
I20260926 16:35:46.687115 25038 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:35:46.689709 25038 webserver.cc:460] Webserver started at http://127.6.33.62:45293/ using document root <none> and password file <none>
I20260926 16:35:46.690444 25038 fs_manager.cc:359] Metadata directory not provided
I20260926 16:35:46.690563 25038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:35:46.690905 25038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:35:46.694028 25038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/master-0/data/instance:
uuid: "b8bb1969c8d54a40bd716611dffffc6a"
format_stamp: "Formatted at 2026-09-26 16:35:46 on dist-test-slave-khd3"
server_key: "ffb691f8e02e430232a30f9f14194e53"
server_key_iv: "bc33c352c54d5fbb650ac99cb24e2cca"
server_key_version: "encryptionkey@0"
I20260926 16:35:46.694864 25038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/master-0/wal/instance:
uuid: "b8bb1969c8d54a40bd716611dffffc6a"
format_stamp: "Formatted at 2026-09-26 16:35:46 on dist-test-slave-khd3"
server_key: "ffb691f8e02e430232a30f9f14194e53"
server_key_iv: "bc33c352c54d5fbb650ac99cb24e2cca"
server_key_version: "encryptionkey@0"
I20260926 16:35:46.700407 25038 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260926 16:35:46.704257 25052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:35:46.705806 25038 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:35:46.706022 25038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/master-0/wal
uuid: "b8bb1969c8d54a40bd716611dffffc6a"
format_stamp: "Formatted at 2026-09-26 16:35:46 on dist-test-slave-khd3"
server_key: "ffb691f8e02e430232a30f9f14194e53"
server_key_iv: "bc33c352c54d5fbb650ac99cb24e2cca"
server_key_version: "encryptionkey@0"
I20260926 16:35:46.706272 25038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:35:46.722092 25038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:35:46.723029 25038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:35:46.723347 25038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:35:46.742525 25038 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.62:44035
I20260926 16:35:46.742548 25103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.62:44035 every 8 connection(s)
I20260926 16:35:46.743997 25038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
I20260926 16:35:46.747996 25104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:35:46.750422 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 25038
I20260926 16:35:46.750614 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/master-0/wal/instance
I20260926 16:35:46.751091 6276 external_mini_cluster.cc:1428] Setting key d59cbbd2ca046928188925b53e336479
I20260926 16:35:46.759558 25104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8bb1969c8d54a40bd716611dffffc6a: Bootstrap starting.
I20260926 16:35:46.762863 25104 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8bb1969c8d54a40bd716611dffffc6a: Neither blocks nor log segments found. Creating new log.
I20260926 16:35:46.764019 25104 log.cc:824] T 00000000000000000000000000000000 P b8bb1969c8d54a40bd716611dffffc6a: Log is configured to *not* fsync() on all Append() calls
I20260926 16:35:46.764706 25107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:46.753011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52998 (local address 127.6.33.62:44035)
0926 16:35:46.753996 (+ 985us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:46.754007 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:46.754048 (+ 41us) server_negotiation.cc:407] Connection header received
0926 16:35:46.755740 (+ 1692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:46.755762 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:46.756228 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:46.756646 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:35:46.757295 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:46.758594 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:46.760054 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:46.760733 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:46.761013 (+ 280us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:46.762244 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:35:46.762251 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:35:46.762264 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:35:46.762781 (+ 517us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:35:46.762882 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:35:46.762903 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:35:46.763100 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:46.763334 (+ 234us) server_negotiation.cc:299] Negotiation successful
0926 16:35:46.763637 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":346,"threads_started":1}
I20260926 16:35:46.767040 25104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8bb1969c8d54a40bd716611dffffc6a: No bootstrap required, opened a new log
I20260926 16:35:46.774492 25104 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8bb1969c8d54a40bd716611dffffc6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8bb1969c8d54a40bd716611dffffc6a" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 44035 } }
I20260926 16:35:46.774894 25104 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8bb1969c8d54a40bd716611dffffc6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:35:46.775012 25104 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8bb1969c8d54a40bd716611dffffc6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8bb1969c8d54a40bd716611dffffc6a, State: Initialized, Role: FOLLOWER
I20260926 16:35:46.775645 25104 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8bb1969c8d54a40bd716611dffffc6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8bb1969c8d54a40bd716611dffffc6a" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 44035 } }
I20260926 16:35:46.775895 25104 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8bb1969c8d54a40bd716611dffffc6a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:35:46.776041 25104 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8bb1969c8d54a40bd716611dffffc6a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:35:46.776226 25104 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8bb1969c8d54a40bd716611dffffc6a [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:35:46.778610 25104 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8bb1969c8d54a40bd716611dffffc6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8bb1969c8d54a40bd716611dffffc6a" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 44035 } }
I20260926 16:35:46.779237 25104 leader_election.cc:302] T 00000000000000000000000000000000 P b8bb1969c8d54a40bd716611dffffc6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8bb1969c8d54a40bd716611dffffc6a; no voters:
I20260926 16:35:46.779836 25104 leader_election.cc:288] T 00000000000000000000000000000000 P b8bb1969c8d54a40bd716611dffffc6a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:35:46.779990 25109 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8bb1969c8d54a40bd716611dffffc6a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:35:46.780822 25109 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8bb1969c8d54a40bd716611dffffc6a [term 1 LEADER]: Becoming Leader. State: Replica: b8bb1969c8d54a40bd716611dffffc6a, State: Running, Role: LEADER
I20260926 16:35:46.781440 25109 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8bb1969c8d54a40bd716611dffffc6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8bb1969c8d54a40bd716611dffffc6a" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 44035 } }
I20260926 16:35:46.782052 25104 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8bb1969c8d54a40bd716611dffffc6a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:35:46.784348 25111 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8bb1969c8d54a40bd716611dffffc6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8bb1969c8d54a40bd716611dffffc6a. Latest consensus state: current_term: 1 leader_uuid: "b8bb1969c8d54a40bd716611dffffc6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8bb1969c8d54a40bd716611dffffc6a" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 44035 } } }
I20260926 16:35:46.784912 25111 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8bb1969c8d54a40bd716611dffffc6a [sys.catalog]: This master's current role is: LEADER
I20260926 16:35:46.784432 25110 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8bb1969c8d54a40bd716611dffffc6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8bb1969c8d54a40bd716611dffffc6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8bb1969c8d54a40bd716611dffffc6a" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 44035 } } }
I20260926 16:35:46.785168 25110 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8bb1969c8d54a40bd716611dffffc6a [sys.catalog]: This master's current role is: LEADER
I20260926 16:35:46.786775 25115 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:35:46.791817 25115 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:35:46.801709 25115 catalog_manager.cc:1347] Generated new cluster ID: abf41162bcda471d919e9ece8c461a68
I20260926 16:35:46.801827 25115 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:35:46.815128 25115 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:35:46.817057 25115 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:35:46.826735 25115 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8bb1969c8d54a40bd716611dffffc6a: Generated new TSK 0
I20260926 16:35:46.827786 25115 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:35:46.834352 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.1:0
--local_ip_for_outbound_sockets=127.6.33.1
--webserver_interface=127.6.33.1
--webserver_port=0
--tserver_master_addrs=127.6.33.62:44035
--builtin_ntp_servers=127.6.33.20:35547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 16:35:47.175262 25128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:35:47.175595 25128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:35:47.175758 25128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:35:47.183485 25128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:35:47.183629 25128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:35:47.183841 25128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.1
I20260926 16:35:47.193614 25128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:35547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.33.1
--webserver_port=0
--tserver_master_addrs=127.6.33.62:44035
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.1
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:35:47.195017 25128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:35:47.196895 25128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:35:47.210577 25133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:35:47.210649 25134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:35:47.210675 25136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:35:47.211828 25128 server_base.cc:1034] running on GCE node
I20260926 16:35:47.212435 25128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:35:47.213438 25128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:35:47.214684 25128 hybrid_clock.cc:648] HybridClock initialized: now 1790440547214632 us; error 44 us; skew 500 ppm
I20260926 16:35:47.215024 25128 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:35:47.217502 25128 webserver.cc:460] Webserver started at http://127.6.33.1:38753/ using document root <none> and password file <none>
I20260926 16:35:47.218233 25128 fs_manager.cc:359] Metadata directory not provided
I20260926 16:35:47.218369 25128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:35:47.218698 25128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:35:47.221693 25128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/ts-0/data/instance:
uuid: "898ededa37164f23b07fdf61e5eac29b"
format_stamp: "Formatted at 2026-09-26 16:35:47 on dist-test-slave-khd3"
server_key: "12c308eba23754473348349b91c0bbfc"
server_key_iv: "b320e52c893f6715018dc35b70192b4b"
server_key_version: "encryptionkey@0"
I20260926 16:35:47.222532 25128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/ts-0/wal/instance:
uuid: "898ededa37164f23b07fdf61e5eac29b"
format_stamp: "Formatted at 2026-09-26 16:35:47 on dist-test-slave-khd3"
server_key: "12c308eba23754473348349b91c0bbfc"
server_key_iv: "b320e52c893f6715018dc35b70192b4b"
server_key_version: "encryptionkey@0"
I20260926 16:35:47.228097 25128 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 16:35:47.231812 25142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:35:47.233528 25128 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:35:47.233714 25128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/ts-0/wal
uuid: "898ededa37164f23b07fdf61e5eac29b"
format_stamp: "Formatted at 2026-09-26 16:35:47 on dist-test-slave-khd3"
server_key: "12c308eba23754473348349b91c0bbfc"
server_key_iv: "b320e52c893f6715018dc35b70192b4b"
server_key_version: "encryptionkey@0"
I20260926 16:35:47.233948 25128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:35:47.251360 25128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:35:47.252257 25128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:35:47.252548 25128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:35:47.253645 25128 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:35:47.255192 25128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:35:47.255317 25128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:35:47.255455 25128 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:35:47.255542 25128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:35:47.289050 25128 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.1:46175
I20260926 16:35:47.289095 25254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.1:46175 every 8 connection(s)
I20260926 16:35:47.290398 25128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
I20260926 16:35:47.299472 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 25128
I20260926 16:35:47.299662 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/ts-0/wal/instance
I20260926 16:35:47.300087 6276 external_mini_cluster.cc:1428] Setting key 38e922c1881d7e6d19621eb1bbea91d6
I20260926 16:35:47.302735 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.2:0
--local_ip_for_outbound_sockets=127.6.33.2
--webserver_interface=127.6.33.2
--webserver_port=0
--tserver_master_addrs=127.6.33.62:44035
--builtin_ntp_servers=127.6.33.20:35547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 16:35:47.304183 25257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:47.293227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:51987 (local address 127.6.33.62:44035)
0926 16:35:47.293701 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:47.293706 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:47.294878 (+ 1172us) server_negotiation.cc:407] Connection header received
0926 16:35:47.297049 (+ 2171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:47.297053 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:47.297133 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:47.297255 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:35:47.298431 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:47.299203 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:47.301135 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:47.301604 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:47.301746 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:47.303235 (+ 1489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:35:47.303240 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:35:47.303244 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:35:47.303263 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:35:47.303305 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:35:47.303317 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:35:47.303413 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:47.303661 (+ 248us) server_negotiation.cc:299] Negotiation successful
0926 16:35:47.303818 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":256,"threads_started":1}
I20260926 16:35:47.305332 25258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:47.293762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:44035 (local address 127.6.33.1:51987)
0926 16:35:47.294690 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:47.294742 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 16:35:47.296698 (+ 1956us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:35:47.297468 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:35:47.297483 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:35:47.297579 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:35:47.298275 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:47.298293 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:47.299380 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:47.299384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:47.300945 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:47.300958 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:47.301848 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:47.301856 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:47.302099 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:47.302841 (+ 742us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:35:47.302844 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:35:47.302919 (+ 75us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:35:47.303029 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:35:47.303411 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:35:47.303424 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:35:47.303434 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 16:35:47.303638 (+ 204us) client_negotiation.cc:240] Negotiation successful
0926 16:35:47.304102 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":269,"threads_started":1}
I20260926 16:35:47.309321 25255 heartbeater.cc:347] Connected to a master server at 127.6.33.62:44035
I20260926 16:35:47.309731 25255 heartbeater.cc:464] Registering TS with master...
I20260926 16:35:47.310652 25255 heartbeater.cc:511] Master 127.6.33.62:44035 requested a full tablet report, sending...
I20260926 16:35:47.313122 25069 ts_manager.cc:194] Registered new tserver with Master: 898ededa37164f23b07fdf61e5eac29b (127.6.33.1:46175)
I20260926 16:35:47.315408 25069 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.33.1:51987
W20260926 16:35:47.641167 25259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:35:47.641485 25259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:35:47.641584 25259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:35:47.648833 25259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:35:47.648938 25259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:35:47.649045 25259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.2
I20260926 16:35:47.658279 25259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:35547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.33.2
--webserver_port=0
--tserver_master_addrs=127.6.33.62:44035
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.2
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:35:47.659603 25259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:35:47.661518 25259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:35:47.675617 25265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:35:47.675688 25267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:35:47.675711 25264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:35:47.677155 25259 server_base.cc:1034] running on GCE node
I20260926 16:35:47.677830 25259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:35:47.679117 25259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:35:47.680344 25259 hybrid_clock.cc:648] HybridClock initialized: now 1790440547680293 us; error 51 us; skew 500 ppm
I20260926 16:35:47.680651 25259 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:35:47.683202 25259 webserver.cc:460] Webserver started at http://127.6.33.2:45941/ using document root <none> and password file <none>
I20260926 16:35:47.683913 25259 fs_manager.cc:359] Metadata directory not provided
I20260926 16:35:47.684005 25259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:35:47.684270 25259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:35:47.687294 25259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/ts-1/data/instance:
uuid: "2556f3ace30a4eca941815083da1b035"
format_stamp: "Formatted at 2026-09-26 16:35:47 on dist-test-slave-khd3"
server_key: "5702ed6ecf7558ec15ac72578a225fa6"
server_key_iv: "d99e06e1081e2f8c8dcdf010406d07fc"
server_key_version: "encryptionkey@0"
I20260926 16:35:47.688050 25259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/ts-1/wal/instance:
uuid: "2556f3ace30a4eca941815083da1b035"
format_stamp: "Formatted at 2026-09-26 16:35:47 on dist-test-slave-khd3"
server_key: "5702ed6ecf7558ec15ac72578a225fa6"
server_key_iv: "d99e06e1081e2f8c8dcdf010406d07fc"
server_key_version: "encryptionkey@0"
I20260926 16:35:47.693528 25259 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260926 16:35:47.697146 25273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:35:47.698599 25259 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:35:47.698810 25259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/ts-1/wal
uuid: "2556f3ace30a4eca941815083da1b035"
format_stamp: "Formatted at 2026-09-26 16:35:47 on dist-test-slave-khd3"
server_key: "5702ed6ecf7558ec15ac72578a225fa6"
server_key_iv: "d99e06e1081e2f8c8dcdf010406d07fc"
server_key_version: "encryptionkey@0"
I20260926 16:35:47.699069 25259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:35:47.710984 25259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:35:47.711879 25259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:35:47.712117 25259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:35:47.713258 25259 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:35:47.714767 25259 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:35:47.714854 25259 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:35:47.714931 25259 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:35:47.714986 25259 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:35:47.749573 25259 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.2:37319
I20260926 16:35:47.749593 25385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.2:37319 every 8 connection(s)
I20260926 16:35:47.750903 25259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
I20260926 16:35:47.760399 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 25259
I20260926 16:35:47.760566 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/ts-1/wal/instance
I20260926 16:35:47.761117 6276 external_mini_cluster.cc:1428] Setting key 7d28c744e55f72c63f86587da008758c
I20260926 16:35:47.764237 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.3:0
--local_ip_for_outbound_sockets=127.6.33.3
--webserver_interface=127.6.33.3
--webserver_port=0
--tserver_master_addrs=127.6.33.62:44035
--builtin_ntp_servers=127.6.33.20:35547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 16:35:47.766062 25257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:47.753738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:52177 (local address 127.6.33.62:44035)
0926 16:35:47.753965 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:47.753972 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:47.755668 (+ 1696us) server_negotiation.cc:407] Connection header received
0926 16:35:47.758290 (+ 2622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:47.758294 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:47.758366 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:47.758459 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:35:47.759838 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:47.760707 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:47.762722 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:47.763320 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:47.763457 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:47.765119 (+ 1662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:35:47.765128 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:35:47.765137 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:35:47.765164 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:35:47.765211 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:35:47.765230 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:35:47.765334 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:47.765549 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 16:35:47.765660 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 16:35:47.766973 25388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:47.754338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:44035 (local address 127.6.33.2:52177)
0926 16:35:47.755466 (+ 1128us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:47.755511 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 16:35:47.757967 (+ 2456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:35:47.758717 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:35:47.758729 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:35:47.758858 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:35:47.759642 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:47.759666 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:47.760911 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:47.760918 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:47.762584 (+ 1666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:47.762598 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:47.763536 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:47.763541 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:47.763756 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:47.764725 (+ 969us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:35:47.764728 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:35:47.764832 (+ 104us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:35:47.764920 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:35:47.765335 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:35:47.765342 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:35:47.765348 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 16:35:47.765502 (+ 154us) client_negotiation.cc:240] Negotiation successful
0926 16:35:47.765833 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":325,"threads_started":1}
I20260926 16:35:47.768501 25386 heartbeater.cc:347] Connected to a master server at 127.6.33.62:44035
I20260926 16:35:47.768896 25386 heartbeater.cc:464] Registering TS with master...
I20260926 16:35:47.769821 25386 heartbeater.cc:511] Master 127.6.33.62:44035 requested a full tablet report, sending...
I20260926 16:35:47.771484 25069 ts_manager.cc:194] Registered new tserver with Master: 2556f3ace30a4eca941815083da1b035 (127.6.33.2:37319)
I20260926 16:35:47.772400 25069 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.33.2:52177
W20260926 16:35:48.102331 25389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:35:48.102687 25389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:35:48.102861 25389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:35:48.110461 25389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:35:48.110572 25389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:35:48.110745 25389 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.3
I20260926 16:35:48.120096 25389 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:35547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.33.3
--webserver_port=0
--tserver_master_addrs=127.6.33.62:44035
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.3
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:35:48.121523 25389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:35:48.123413 25389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:35:48.136307 25395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:35:48.136386 25394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:35:48.137356 25397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:35:48.137744 25389 server_base.cc:1034] running on GCE node
I20260926 16:35:48.138300 25389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:35:48.139276 25389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:35:48.140489 25389 hybrid_clock.cc:648] HybridClock initialized: now 1790440548140443 us; error 33 us; skew 500 ppm
I20260926 16:35:48.140873 25389 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:35:48.143312 25389 webserver.cc:460] Webserver started at http://127.6.33.3:37953/ using document root <none> and password file <none>
I20260926 16:35:48.144058 25389 fs_manager.cc:359] Metadata directory not provided
I20260926 16:35:48.144183 25389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:35:48.144526 25389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:35:48.147498 25389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/ts-2/data/instance:
uuid: "0d04d60c50b9458284dccfcf209dd7ae"
format_stamp: "Formatted at 2026-09-26 16:35:48 on dist-test-slave-khd3"
server_key: "c3f24f1e85a2447b40540963cc523c29"
server_key_iv: "6c65f9376f4f120f553a81496c673aef"
server_key_version: "encryptionkey@0"
I20260926 16:35:48.148329 25389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/ts-2/wal/instance:
uuid: "0d04d60c50b9458284dccfcf209dd7ae"
format_stamp: "Formatted at 2026-09-26 16:35:48 on dist-test-slave-khd3"
server_key: "c3f24f1e85a2447b40540963cc523c29"
server_key_iv: "6c65f9376f4f120f553a81496c673aef"
server_key_version: "encryptionkey@0"
I20260926 16:35:48.153577 25389 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260926 16:35:48.157294 25403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:35:48.158725 25389 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260926 16:35:48.158910 25389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/ts-2/wal
uuid: "0d04d60c50b9458284dccfcf209dd7ae"
format_stamp: "Formatted at 2026-09-26 16:35:48 on dist-test-slave-khd3"
server_key: "c3f24f1e85a2447b40540963cc523c29"
server_key_iv: "6c65f9376f4f120f553a81496c673aef"
server_key_version: "encryptionkey@0"
I20260926 16:35:48.159142 25389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:35:48.182061 25389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:35:48.183074 25389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:35:48.183399 25389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:35:48.184456 25389 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:35:48.186080 25389 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:35:48.186241 25389 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:35:48.186379 25389 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:35:48.186465 25389 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:35:48.222015 25389 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.3:39593
I20260926 16:35:48.222054 25515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.3:39593 every 8 connection(s)
I20260926 16:35:48.223399 25389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
I20260926 16:35:48.232197 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 25389
I20260926 16:35:48.232425 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/ts-2/wal/instance
I20260926 16:35:48.233039 6276 external_mini_cluster.cc:1428] Setting key e9d86534af886e516a7e2349e6781603
I20260926 16:35:48.238003 25257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:48.226142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.3:34419 (local address 127.6.33.62:44035)
0926 16:35:48.226357 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:48.226363 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:48.227922 (+ 1559us) server_negotiation.cc:407] Connection header received
0926 16:35:48.230112 (+ 2190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:48.230116 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:48.230209 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:48.230303 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:35:48.231561 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:48.232305 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:48.234737 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:48.235339 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:48.235495 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:48.237117 (+ 1622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:35:48.237122 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:35:48.237126 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:35:48.237145 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:35:48.237200 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:35:48.237213 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:35:48.237316 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:48.237561 (+ 245us) server_negotiation.cc:299] Negotiation successful
0926 16:35:48.237665 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 16:35:48.239193 25518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:48.226728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:44035 (local address 127.6.33.3:34419)
0926 16:35:48.227733 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:48.227778 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 16:35:48.229811 (+ 2033us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:35:48.230570 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:35:48.230580 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:35:48.230684 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:35:48.231391 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:48.231411 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:48.232514 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:48.232519 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:48.234566 (+ 2047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:48.234584 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:48.235599 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:48.235607 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:48.235870 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:48.236668 (+ 798us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:35:48.236686 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:35:48.236733 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:35:48.236889 (+ 156us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:35:48.237316 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:35:48.237330 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:35:48.237341 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 16:35:48.237534 (+ 193us) client_negotiation.cc:240] Negotiation successful
0926 16:35:48.238001 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":318,"threads_started":1}
I20260926 16:35:48.240742 25516 heartbeater.cc:347] Connected to a master server at 127.6.33.62:44035
I20260926 16:35:48.241115 25516 heartbeater.cc:464] Registering TS with master...
I20260926 16:35:48.241930 25516 heartbeater.cc:511] Master 127.6.33.62:44035 requested a full tablet report, sending...
I20260926 16:35:48.243441 25069 ts_manager.cc:194] Registered new tserver with Master: 0d04d60c50b9458284dccfcf209dd7ae (127.6.33.3:39593)
I20260926 16:35:48.244271 25069 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.33.3:34419
I20260926 16:35:48.248916 6276 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:35:48.259586 25257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:48.253092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53002 (local address 127.6.33.62:44035)
0926 16:35:48.253309 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:48.253314 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:48.253914 (+ 600us) server_negotiation.cc:407] Connection header received
0926 16:35:48.254080 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:48.254084 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:48.254168 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:48.254263 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:35:48.254728 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:48.255532 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:48.257025 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:48.257642 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:48.257753 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:48.258864 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:35:48.258868 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:35:48.258872 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:35:48.258890 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:35:48.258923 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:35:48.258935 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:35:48.259118 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:48.259299 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 16:35:48.259345 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 16:35:48.263923 25069 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53002:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:35:48.270815 25069 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:35:48.311785 25530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:48.284187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:46175 (local address 127.0.0.1:39102)
0926 16:35:48.285904 (+ 1717us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:48.285927 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 16:35:48.286031 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:35:48.304237 (+ 18206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:35:48.304248 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:35:48.304270 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:35:48.304620 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:48.304629 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:48.306802 (+ 2173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:48.306807 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:48.308933 (+ 2126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:48.308946 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:48.310702 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:48.310709 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:48.310958 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:48.311039 (+ 81us) client_negotiation.cc:769] Sending connection context
0926 16:35:48.311174 (+ 135us) client_negotiation.cc:240] Negotiation successful
0926 16:35:48.311361 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1574,"mutex_wait_us":22,"thread_start_us":1438,"threads_started":1}
I20260926 16:35:48.312147 25533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:48.285852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38922 (local address 127.6.33.2:37319)
0926 16:35:48.304226 (+ 18374us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:48.304234 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:48.304261 (+ 27us) server_negotiation.cc:407] Connection header received
0926 16:35:48.304375 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:48.304384 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:48.304600 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:48.304806 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:35:48.305504 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:48.306453 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:48.309761 (+ 3308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:48.310978 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:48.311113 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:48.311351 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:48.311611 (+ 260us) server_negotiation.cc:299] Negotiation successful
0926 16:35:48.311814 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18207,"thread_start_us":18041,"threads_started":1}
I20260926 16:35:48.312150 25528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:48.283838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.3:39593 (local address 127.0.0.1:48438)
0926 16:35:48.284901 (+ 1063us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:48.284963 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 16:35:48.285188 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:35:48.304002 (+ 18814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:35:48.304012 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:35:48.304044 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:35:48.304565 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:48.304583 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:48.306673 (+ 2090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:48.306679 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:48.309530 (+ 2851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:48.309557 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:48.311428 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:48.311433 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:48.311617 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:48.311644 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 16:35:48.311697 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 16:35:48.311826 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"mutex_wait_us":313,"thread_start_us":501,"threads_started":1}
I20260926 16:35:48.312469 25531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:48.284708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39102 (local address 127.6.33.1:46175)
0926 16:35:48.302025 (+ 17317us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:48.302036 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:48.302072 (+ 36us) server_negotiation.cc:407] Connection header received
0926 16:35:48.302195 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:48.302205 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:48.302505 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:48.304856 (+ 2351us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:35:48.305117 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:48.306644 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:48.309122 (+ 2478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:48.310504 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:48.310639 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:48.310923 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:48.311915 (+ 992us) server_negotiation.cc:299] Negotiation successful
0926 16:35:48.312102 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17114,"thread_start_us":16980,"threads_started":1}
I20260926 16:35:48.312781 25532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:48.285304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.2:37319 (local address 127.0.0.1:38922)
0926 16:35:48.303588 (+ 18284us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:48.303610 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 16:35:48.303710 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:35:48.305009 (+ 1299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:35:48.305012 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:35:48.305030 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:35:48.305351 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:48.305360 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:48.306599 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:48.306606 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:48.309529 (+ 2923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:48.309544 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:48.311177 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:48.311184 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:48.311399 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:48.311439 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 16:35:48.311520 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 16:35:48.311646 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18156,"mutex_wait_us":347,"thread_start_us":18018,"threads_started":1}
I20260926 16:35:48.313308 25529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:48.283991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48438 (local address 127.6.33.3:39593)
0926 16:35:48.303202 (+ 19211us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:48.303214 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:48.303260 (+ 46us) server_negotiation.cc:407] Connection header received
0926 16:35:48.303381 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:48.303394 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:48.303723 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:48.303956 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:35:48.304836 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:48.306513 (+ 1677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:48.309840 (+ 3327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:48.311217 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:48.312400 (+ 1183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:48.312703 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:48.312798 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 16:35:48.312975 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18995,"thread_start_us":18910,"threads_started":1}
I20260926 16:35:48.316573 25190 tablet_service.cc:1467] Processing CreateTablet for tablet 3987aca406354cbfab4191de24cb5c62 (DEFAULT_TABLE table=test-table [id=f77b6adefda44f6a843d13146c1f55e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:35:48.316583 25321 tablet_service.cc:1467] Processing CreateTablet for tablet 3987aca406354cbfab4191de24cb5c62 (DEFAULT_TABLE table=test-table [id=f77b6adefda44f6a843d13146c1f55e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:35:48.316584 25451 tablet_service.cc:1467] Processing CreateTablet for tablet 3987aca406354cbfab4191de24cb5c62 (DEFAULT_TABLE table=test-table [id=f77b6adefda44f6a843d13146c1f55e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:35:48.317901 25190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3987aca406354cbfab4191de24cb5c62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:35:48.317948 25451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3987aca406354cbfab4191de24cb5c62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:35:48.317948 25321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3987aca406354cbfab4191de24cb5c62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:35:48.318720 25255 heartbeater.cc:503] Master 127.6.33.62:44035 was elected leader, sending a full tablet report...
I20260926 16:35:48.329365 25534 tablet_bootstrap.cc:492] T 3987aca406354cbfab4191de24cb5c62 P 898ededa37164f23b07fdf61e5eac29b: Bootstrap starting.
I20260926 16:35:48.332549 25535 tablet_bootstrap.cc:492] T 3987aca406354cbfab4191de24cb5c62 P 2556f3ace30a4eca941815083da1b035: Bootstrap starting.
I20260926 16:35:48.333280 25534 tablet_bootstrap.cc:654] T 3987aca406354cbfab4191de24cb5c62 P 898ededa37164f23b07fdf61e5eac29b: Neither blocks nor log segments found. Creating new log.
I20260926 16:35:48.334705 25534 log.cc:824] T 3987aca406354cbfab4191de24cb5c62 P 898ededa37164f23b07fdf61e5eac29b: Log is configured to *not* fsync() on all Append() calls
I20260926 16:35:48.336889 25535 tablet_bootstrap.cc:654] T 3987aca406354cbfab4191de24cb5c62 P 2556f3ace30a4eca941815083da1b035: Neither blocks nor log segments found. Creating new log.
I20260926 16:35:48.338801 25535 log.cc:824] T 3987aca406354cbfab4191de24cb5c62 P 2556f3ace30a4eca941815083da1b035: Log is configured to *not* fsync() on all Append() calls
I20260926 16:35:48.339661 25536 tablet_bootstrap.cc:492] T 3987aca406354cbfab4191de24cb5c62 P 0d04d60c50b9458284dccfcf209dd7ae: Bootstrap starting.
I20260926 16:35:48.340072 25534 tablet_bootstrap.cc:492] T 3987aca406354cbfab4191de24cb5c62 P 898ededa37164f23b07fdf61e5eac29b: No bootstrap required, opened a new log
I20260926 16:35:48.340375 25534 ts_tablet_manager.cc:1397] T 3987aca406354cbfab4191de24cb5c62 P 898ededa37164f23b07fdf61e5eac29b: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20260926 16:35:48.341650 25535 tablet_bootstrap.cc:492] T 3987aca406354cbfab4191de24cb5c62 P 2556f3ace30a4eca941815083da1b035: No bootstrap required, opened a new log
I20260926 16:35:48.341922 25535 ts_tablet_manager.cc:1397] T 3987aca406354cbfab4191de24cb5c62 P 2556f3ace30a4eca941815083da1b035: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260926 16:35:48.343600 25536 tablet_bootstrap.cc:654] T 3987aca406354cbfab4191de24cb5c62 P 0d04d60c50b9458284dccfcf209dd7ae: Neither blocks nor log segments found. Creating new log.
I20260926 16:35:48.345048 25536 log.cc:824] T 3987aca406354cbfab4191de24cb5c62 P 0d04d60c50b9458284dccfcf209dd7ae: Log is configured to *not* fsync() on all Append() calls
I20260926 16:35:48.348686 25536 tablet_bootstrap.cc:492] T 3987aca406354cbfab4191de24cb5c62 P 0d04d60c50b9458284dccfcf209dd7ae: No bootstrap required, opened a new log
I20260926 16:35:48.349116 25536 ts_tablet_manager.cc:1397] T 3987aca406354cbfab4191de24cb5c62 P 0d04d60c50b9458284dccfcf209dd7ae: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20260926 16:35:48.350283 25535 raft_consensus.cc:348] T 3987aca406354cbfab4191de24cb5c62 P 2556f3ace30a4eca941815083da1b035 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d04d60c50b9458284dccfcf209dd7ae" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39593 } } peers { permanent_uuid: "2556f3ace30a4eca941815083da1b035" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37319 } } peers { permanent_uuid: "898ededa37164f23b07fdf61e5eac29b" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 46175 } }
I20260926 16:35:48.350696 25535 raft_consensus.cc:374] T 3987aca406354cbfab4191de24cb5c62 P 2556f3ace30a4eca941815083da1b035 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:35:48.350795 25535 raft_consensus.cc:729] T 3987aca406354cbfab4191de24cb5c62 P 2556f3ace30a4eca941815083da1b035 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2556f3ace30a4eca941815083da1b035, State: Initialized, Role: FOLLOWER
I20260926 16:35:48.351003 25534 raft_consensus.cc:348] T 3987aca406354cbfab4191de24cb5c62 P 898ededa37164f23b07fdf61e5eac29b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d04d60c50b9458284dccfcf209dd7ae" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39593 } } peers { permanent_uuid: "2556f3ace30a4eca941815083da1b035" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37319 } } peers { permanent_uuid: "898ededa37164f23b07fdf61e5eac29b" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 46175 } }
I20260926 16:35:48.351452 25534 raft_consensus.cc:374] T 3987aca406354cbfab4191de24cb5c62 P 898ededa37164f23b07fdf61e5eac29b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:35:48.351559 25534 raft_consensus.cc:729] T 3987aca406354cbfab4191de24cb5c62 P 898ededa37164f23b07fdf61e5eac29b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 898ededa37164f23b07fdf61e5eac29b, State: Initialized, Role: FOLLOWER
I20260926 16:35:48.351519 25535 consensus_queue.cc:260] T 3987aca406354cbfab4191de24cb5c62 P 2556f3ace30a4eca941815083da1b035 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d04d60c50b9458284dccfcf209dd7ae" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39593 } } peers { permanent_uuid: "2556f3ace30a4eca941815083da1b035" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37319 } } peers { permanent_uuid: "898ededa37164f23b07fdf61e5eac29b" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 46175 } }
I20260926 16:35:48.352279 25534 consensus_queue.cc:260] T 3987aca406354cbfab4191de24cb5c62 P 898ededa37164f23b07fdf61e5eac29b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d04d60c50b9458284dccfcf209dd7ae" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39593 } } peers { permanent_uuid: "2556f3ace30a4eca941815083da1b035" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37319 } } peers { permanent_uuid: "898ededa37164f23b07fdf61e5eac29b" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 46175 } }
I20260926 16:35:48.352847 25386 heartbeater.cc:503] Master 127.6.33.62:44035 was elected leader, sending a full tablet report...
I20260926 16:35:48.353431 25535 ts_tablet_manager.cc:1428] T 3987aca406354cbfab4191de24cb5c62 P 2556f3ace30a4eca941815083da1b035: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260926 16:35:48.355599 25534 ts_tablet_manager.cc:1428] T 3987aca406354cbfab4191de24cb5c62 P 898ededa37164f23b07fdf61e5eac29b: Time spent starting tablet: real 0.015s user 0.009s sys 0.004s
I20260926 16:35:48.359342 25536 raft_consensus.cc:348] T 3987aca406354cbfab4191de24cb5c62 P 0d04d60c50b9458284dccfcf209dd7ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d04d60c50b9458284dccfcf209dd7ae" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39593 } } peers { permanent_uuid: "2556f3ace30a4eca941815083da1b035" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37319 } } peers { permanent_uuid: "898ededa37164f23b07fdf61e5eac29b" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 46175 } }
I20260926 16:35:48.359779 25536 raft_consensus.cc:374] T 3987aca406354cbfab4191de24cb5c62 P 0d04d60c50b9458284dccfcf209dd7ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:35:48.359881 25536 raft_consensus.cc:729] T 3987aca406354cbfab4191de24cb5c62 P 0d04d60c50b9458284dccfcf209dd7ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d04d60c50b9458284dccfcf209dd7ae, State: Initialized, Role: FOLLOWER
I20260926 16:35:48.360517 25536 consensus_queue.cc:260] T 3987aca406354cbfab4191de24cb5c62 P 0d04d60c50b9458284dccfcf209dd7ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d04d60c50b9458284dccfcf209dd7ae" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39593 } } peers { permanent_uuid: "2556f3ace30a4eca941815083da1b035" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37319 } } peers { permanent_uuid: "898ededa37164f23b07fdf61e5eac29b" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 46175 } }
I20260926 16:35:48.361532 25516 heartbeater.cc:503] Master 127.6.33.62:44035 was elected leader, sending a full tablet report...
I20260926 16:35:48.362007 25536 ts_tablet_manager.cc:1428] T 3987aca406354cbfab4191de24cb5c62 P 0d04d60c50b9458284dccfcf209dd7ae: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260926 16:35:48.385201 25542 raft_consensus.cc:482] T 3987aca406354cbfab4191de24cb5c62 P 0d04d60c50b9458284dccfcf209dd7ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:35:48.385543 25542 raft_consensus.cc:504] T 3987aca406354cbfab4191de24cb5c62 P 0d04d60c50b9458284dccfcf209dd7ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d04d60c50b9458284dccfcf209dd7ae" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39593 } } peers { permanent_uuid: "2556f3ace30a4eca941815083da1b035" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37319 } } peers { permanent_uuid: "898ededa37164f23b07fdf61e5eac29b" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 46175 } }
I20260926 16:35:48.387428 25542 leader_election.cc:288] T 3987aca406354cbfab4191de24cb5c62 P 0d04d60c50b9458284dccfcf209dd7ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2556f3ace30a4eca941815083da1b035 (127.6.33.2:37319), 898ededa37164f23b07fdf61e5eac29b (127.6.33.1:46175)
I20260926 16:35:48.394788 25533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:48.387822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.3:44701 (local address 127.6.33.2:37319)
0926 16:35:48.388037 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:48.388044 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:48.388162 (+ 118us) server_negotiation.cc:407] Connection header received
0926 16:35:48.388432 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:48.388440 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:48.388549 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:48.388652 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:35:48.389207 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:48.390209 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:48.392711 (+ 2502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:48.393950 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:48.394088 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:48.394186 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:48.394432 (+ 246us) server_negotiation.cc:299] Negotiation successful
0926 16:35:48.394575 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 16:35:48.394897 25518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:48.387689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.2:37319 (local address 127.6.33.3:44701)
0926 16:35:48.388035 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:48.388055 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 16:35:48.388227 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:35:48.388723 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:35:48.388729 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:35:48.388787 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:35:48.389027 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:48.389045 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:48.390342 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:48.390345 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:48.392532 (+ 2187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:48.392548 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:48.394133 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:48.394138 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:48.394263 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:48.394289 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 16:35:48.394367 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 16:35:48.394539 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":10,"spinlock_wait_cycles":21888}
I20260926 16:35:48.395309 25531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:48.387964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.3:39667 (local address 127.6.33.1:46175)
0926 16:35:48.388197 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:48.388207 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:48.388840 (+ 633us) server_negotiation.cc:407] Connection header received
0926 16:35:48.389084 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:48.389088 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:48.389224 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:48.389330 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:35:48.389798 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:48.390660 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:48.393132 (+ 2472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:48.394332 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:48.394481 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:48.394570 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:48.394916 (+ 346us) server_negotiation.cc:299] Negotiation successful
0926 16:35:48.395061 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 16:35:48.395641 25543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:48.387853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:46175 (local address 127.6.33.3:39667)
0926 16:35:48.388723 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:48.388739 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:35:48.388902 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:35:48.389351 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:35:48.389354 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:35:48.389375 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:35:48.389632 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:48.389639 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:48.390806 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:48.390817 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:48.392914 (+ 2097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:48.392925 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:48.394555 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:48.394561 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:48.394741 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:48.394773 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 16:35:48.395105 (+ 332us) client_negotiation.cc:240] Negotiation successful
0926 16:35:48.395256 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"mutex_wait_us":27,"spinlock_wait_cycles":4224,"thread_start_us":479,"threads_started":1}
I20260926 16:35:48.396363 25341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3987aca406354cbfab4191de24cb5c62" candidate_uuid: "0d04d60c50b9458284dccfcf209dd7ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2556f3ace30a4eca941815083da1b035" is_pre_election: true
I20260926 16:35:48.396941 25341 raft_consensus.cc:2454] T 3987aca406354cbfab4191de24cb5c62 P 2556f3ace30a4eca941815083da1b035 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d04d60c50b9458284dccfcf209dd7ae in term 0.
I20260926 16:35:48.396842 25210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3987aca406354cbfab4191de24cb5c62" candidate_uuid: "0d04d60c50b9458284dccfcf209dd7ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "898ededa37164f23b07fdf61e5eac29b" is_pre_election: true
I20260926 16:35:48.397424 25210 raft_consensus.cc:2454] T 3987aca406354cbfab4191de24cb5c62 P 898ededa37164f23b07fdf61e5eac29b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d04d60c50b9458284dccfcf209dd7ae in term 0.
I20260926 16:35:48.397728 25404 leader_election.cc:302] T 3987aca406354cbfab4191de24cb5c62 P 0d04d60c50b9458284dccfcf209dd7ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d04d60c50b9458284dccfcf209dd7ae, 2556f3ace30a4eca941815083da1b035; no voters:
I20260926 16:35:48.398192 25542 raft_consensus.cc:2781] T 3987aca406354cbfab4191de24cb5c62 P 0d04d60c50b9458284dccfcf209dd7ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:35:48.398346 25542 raft_consensus.cc:482] T 3987aca406354cbfab4191de24cb5c62 P 0d04d60c50b9458284dccfcf209dd7ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:35:48.398454 25542 raft_consensus.cc:3037] T 3987aca406354cbfab4191de24cb5c62 P 0d04d60c50b9458284dccfcf209dd7ae [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:35:48.400709 25542 raft_consensus.cc:504] T 3987aca406354cbfab4191de24cb5c62 P 0d04d60c50b9458284dccfcf209dd7ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d04d60c50b9458284dccfcf209dd7ae" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39593 } } peers { permanent_uuid: "2556f3ace30a4eca941815083da1b035" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37319 } } peers { permanent_uuid: "898ededa37164f23b07fdf61e5eac29b" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 46175 } }
I20260926 16:35:48.401393 25542 leader_election.cc:288] T 3987aca406354cbfab4191de24cb5c62 P 0d04d60c50b9458284dccfcf209dd7ae [CANDIDATE]: Term 1 election: Requested vote from peers 2556f3ace30a4eca941815083da1b035 (127.6.33.2:37319), 898ededa37164f23b07fdf61e5eac29b (127.6.33.1:46175)
I20260926 16:35:48.401870 25341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3987aca406354cbfab4191de24cb5c62" candidate_uuid: "0d04d60c50b9458284dccfcf209dd7ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2556f3ace30a4eca941815083da1b035"
I20260926 16:35:48.401953 25210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3987aca406354cbfab4191de24cb5c62" candidate_uuid: "0d04d60c50b9458284dccfcf209dd7ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "898ededa37164f23b07fdf61e5eac29b"
I20260926 16:35:48.402082 25341 raft_consensus.cc:3037] T 3987aca406354cbfab4191de24cb5c62 P 2556f3ace30a4eca941815083da1b035 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:35:48.402194 25210 raft_consensus.cc:3037] T 3987aca406354cbfab4191de24cb5c62 P 898ededa37164f23b07fdf61e5eac29b [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:35:48.404582 25210 raft_consensus.cc:2454] T 3987aca406354cbfab4191de24cb5c62 P 898ededa37164f23b07fdf61e5eac29b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d04d60c50b9458284dccfcf209dd7ae in term 1.
I20260926 16:35:48.404664 25341 raft_consensus.cc:2454] T 3987aca406354cbfab4191de24cb5c62 P 2556f3ace30a4eca941815083da1b035 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d04d60c50b9458284dccfcf209dd7ae in term 1.
I20260926 16:35:48.405053 25406 leader_election.cc:302] T 3987aca406354cbfab4191de24cb5c62 P 0d04d60c50b9458284dccfcf209dd7ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d04d60c50b9458284dccfcf209dd7ae, 898ededa37164f23b07fdf61e5eac29b; no voters:
I20260926 16:35:48.405418 25542 raft_consensus.cc:2781] T 3987aca406354cbfab4191de24cb5c62 P 0d04d60c50b9458284dccfcf209dd7ae [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:35:48.405856 25542 raft_consensus.cc:686] T 3987aca406354cbfab4191de24cb5c62 P 0d04d60c50b9458284dccfcf209dd7ae [term 1 LEADER]: Becoming Leader. State: Replica: 0d04d60c50b9458284dccfcf209dd7ae, State: Running, Role: LEADER
I20260926 16:35:48.406333 25542 consensus_queue.cc:237] T 3987aca406354cbfab4191de24cb5c62 P 0d04d60c50b9458284dccfcf209dd7ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d04d60c50b9458284dccfcf209dd7ae" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39593 } } peers { permanent_uuid: "2556f3ace30a4eca941815083da1b035" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37319 } } peers { permanent_uuid: "898ededa37164f23b07fdf61e5eac29b" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 46175 } }
I20260926 16:35:48.411386 25069 catalog_manager.cc:5495] T 3987aca406354cbfab4191de24cb5c62 P 0d04d60c50b9458284dccfcf209dd7ae reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d04d60c50b9458284dccfcf209dd7ae (127.6.33.3). New cstate: current_term: 1 leader_uuid: "0d04d60c50b9458284dccfcf209dd7ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d04d60c50b9458284dccfcf209dd7ae" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 39593 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2556f3ace30a4eca941815083da1b035" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "898ededa37164f23b07fdf61e5eac29b" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 46175 } health_report { overall_health: UNKNOWN } } }
I20260926 16:35:48.453106 25529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:48.447419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48452 (local address 127.6.33.3:39593)
0926 16:35:48.447630 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:48.447634 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:48.447653 (+ 19us) server_negotiation.cc:407] Connection header received
0926 16:35:48.447774 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:48.447778 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:48.447855 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:48.447982 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:35:48.448382 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:48.449196 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:48.450699 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:48.451110 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:48.451206 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:48.451850 (+ 644us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:35:48.452360 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:35:48.452499 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:48.452727 (+ 228us) server_negotiation.cc:299] Negotiation successful
0926 16:35:48.452824 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 16:35:48.461755 25210 raft_consensus.cc:1261] T 3987aca406354cbfab4191de24cb5c62 P 898ededa37164f23b07fdf61e5eac29b [term 1 FOLLOWER]: Refusing update from remote peer 0d04d60c50b9458284dccfcf209dd7ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:35:48.461753 25341 raft_consensus.cc:1261] T 3987aca406354cbfab4191de24cb5c62 P 2556f3ace30a4eca941815083da1b035 [term 1 FOLLOWER]: Refusing update from remote peer 0d04d60c50b9458284dccfcf209dd7ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:35:48.462772 25544 consensus_queue.cc:1035] T 3987aca406354cbfab4191de24cb5c62 P 0d04d60c50b9458284dccfcf209dd7ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "2556f3ace30a4eca941815083da1b035" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 37319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:35:48.463161 25542 consensus_queue.cc:1035] T 3987aca406354cbfab4191de24cb5c62 P 0d04d60c50b9458284dccfcf209dd7ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "898ededa37164f23b07fdf61e5eac29b" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 46175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:35:48.494853 25068 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53002:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:35:48.495239 25068 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53002:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:35:48.498966 25068 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b8bb1969c8d54a40bd716611dffffc6a: Sending DeleteTablet for 3 replicas of tablet 3987aca406354cbfab4191de24cb5c62
I20260926 16:35:48.500175 25451 tablet_service.cc:1514] Processing DeleteTablet for tablet 3987aca406354cbfab4191de24cb5c62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:35:48 UTC) from {username='slave'} at 127.0.0.1:48438
I20260926 16:35:48.500288 25321 tablet_service.cc:1514] Processing DeleteTablet for tablet 3987aca406354cbfab4191de24cb5c62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:35:48 UTC) from {username='slave'} at 127.0.0.1:38922
I20260926 16:35:48.500952 25190 tablet_service.cc:1514] Processing DeleteTablet for tablet 3987aca406354cbfab4191de24cb5c62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:35:48 UTC) from {username='slave'} at 127.0.0.1:39102
I20260926 16:35:48.502111 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 25128
I20260926 16:35:48.510913 25558 tablet_replica.cc:330] T 3987aca406354cbfab4191de24cb5c62 P 2556f3ace30a4eca941815083da1b035: stopping tablet replica
I20260926 16:35:48.511554 25558 raft_consensus.cc:2229] T 3987aca406354cbfab4191de24cb5c62 P 2556f3ace30a4eca941815083da1b035 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:35:48.512262 25558 raft_consensus.cc:2258] T 3987aca406354cbfab4191de24cb5c62 P 2556f3ace30a4eca941815083da1b035 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:35:48.519526 25558 ts_tablet_manager.cc:1905] T 3987aca406354cbfab4191de24cb5c62 P 2556f3ace30a4eca941815083da1b035: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:35:48.524359 25559 tablet_replica.cc:330] T 3987aca406354cbfab4191de24cb5c62 P 898ededa37164f23b07fdf61e5eac29b: stopping tablet replica
I20260926 16:35:48.524922 25559 raft_consensus.cc:2229] T 3987aca406354cbfab4191de24cb5c62 P 898ededa37164f23b07fdf61e5eac29b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:35:48.525497 25559 raft_consensus.cc:2258] T 3987aca406354cbfab4191de24cb5c62 P 898ededa37164f23b07fdf61e5eac29b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:35:48.526453 25557 tablet_replica.cc:330] T 3987aca406354cbfab4191de24cb5c62 P 0d04d60c50b9458284dccfcf209dd7ae: stopping tablet replica
I20260926 16:35:48.526908 25557 raft_consensus.cc:2229] T 3987aca406354cbfab4191de24cb5c62 P 0d04d60c50b9458284dccfcf209dd7ae [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:35:48.528059 25557 raft_consensus.cc:2258] T 3987aca406354cbfab4191de24cb5c62 P 0d04d60c50b9458284dccfcf209dd7ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:35:48.528614 25558 ts_tablet_manager.cc:1918] T 3987aca406354cbfab4191de24cb5c62 P 2556f3ace30a4eca941815083da1b035: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:35:48.528853 25558 log.cc:1196] T 3987aca406354cbfab4191de24cb5c62 P 2556f3ace30a4eca941815083da1b035: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/ts-1/wal/wals/3987aca406354cbfab4191de24cb5c62
I20260926 16:35:48.528939 25559 ts_tablet_manager.cc:1905] T 3987aca406354cbfab4191de24cb5c62 P 898ededa37164f23b07fdf61e5eac29b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:35:48.529533 25558 ts_tablet_manager.cc:1939] T 3987aca406354cbfab4191de24cb5c62 P 2556f3ace30a4eca941815083da1b035: Deleting consensus metadata
I20260926 16:35:48.531860 25053 catalog_manager.cc:4841] TS 2556f3ace30a4eca941815083da1b035 (127.6.33.2:37319): tablet 3987aca406354cbfab4191de24cb5c62 (table test-table [id=f77b6adefda44f6a843d13146c1f55e6]) successfully deleted
I20260926 16:35:48.533762 25531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:48.503773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39110 (local address 127.6.33.1:46175)
0926 16:35:48.504031 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:48.504042 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:48.524060 (+ 20018us) server_negotiation.cc:407] Connection header received
0926 16:35:48.524546 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:48.524554 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:48.524665 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:48.524783 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:35:48.525558 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:48.526812 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:48.529550 (+ 2738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:48.530223 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:48.530326 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:48.531940 (+ 1614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:35:48.531951 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:35:48.531967 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:35:48.532806 (+ 839us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:35:48.532874 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:35:48.532902 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:35:48.533015 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:48.533250 (+ 235us) server_negotiation.cc:299] Negotiation successful
0926 16:35:48.533374 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 16:35:48.534601 25557 ts_tablet_manager.cc:1905] T 3987aca406354cbfab4191de24cb5c62 P 0d04d60c50b9458284dccfcf209dd7ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:35:48.534970 25250 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:35:48.542925 25557 ts_tablet_manager.cc:1918] T 3987aca406354cbfab4191de24cb5c62 P 0d04d60c50b9458284dccfcf209dd7ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:35:48.543144 25557 log.cc:1196] T 3987aca406354cbfab4191de24cb5c62 P 0d04d60c50b9458284dccfcf209dd7ae: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/ts-2/wal/wals/3987aca406354cbfab4191de24cb5c62
I20260926 16:35:48.543707 25557 ts_tablet_manager.cc:1939] T 3987aca406354cbfab4191de24cb5c62 P 0d04d60c50b9458284dccfcf209dd7ae: Deleting consensus metadata
I20260926 16:35:48.545897 25053 catalog_manager.cc:4841] TS 0d04d60c50b9458284dccfcf209dd7ae (127.6.33.3:39593): tablet 3987aca406354cbfab4191de24cb5c62 (table test-table [id=f77b6adefda44f6a843d13146c1f55e6]) successfully deleted
I20260926 16:35:48.627115 25559 ts_tablet_manager.cc:1918] T 3987aca406354cbfab4191de24cb5c62 P 898ededa37164f23b07fdf61e5eac29b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:35:48.627372 25559 log.cc:1196] T 3987aca406354cbfab4191de24cb5c62 P 898ededa37164f23b07fdf61e5eac29b: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440233850748-6276-0/minicluster-data/ts-0/wal/wals/3987aca406354cbfab4191de24cb5c62
I20260926 16:35:48.627871 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 25128
I20260926 16:35:48.651242 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 25259
W20260926 16:35:48.651489 25055 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.33.1:46175 (error 108)
I20260926 16:35:48.654225 25532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:48.653789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:46175 (local address 127.0.0.1:39126)
0926 16:35:48.653964 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:48.654076 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.1:46175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260926 16:35:48.654561 25055 catalog_manager.cc:4584] TS 898ededa37164f23b07fdf61e5eac29b (127.6.33.1:46175): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3987aca406354cbfab4191de24cb5c62: Network error: Client connection negotiation failed: client connection to 127.6.33.1:46175: connect: Connection refused (error 111)
I20260926 16:35:48.659997 25533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:48.652231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38930 (local address 127.6.33.2:37319)
0926 16:35:48.652426 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:48.652432 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:48.652568 (+ 136us) server_negotiation.cc:407] Connection header received
0926 16:35:48.652825 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:48.652832 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:48.652945 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:48.653061 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:35:48.653744 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:48.654887 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:48.656263 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:48.656960 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:48.657093 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:48.658284 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:35:48.658297 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:35:48.658317 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:35:48.659115 (+ 798us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:35:48.659203 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:35:48.659231 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:35:48.659348 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:48.659493 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 16:35:48.659611 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 16:35:48.660929 25381 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:35:48.708355 25532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:48.707832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:46175 (local address 127.0.0.1:39132)
0926 16:35:48.708078 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:48.708187 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.1:46175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260926 16:35:48.709595 25532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:48.709179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:46175 (local address 127.0.0.1:39142)
0926 16:35:48.709347 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:48.709451 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.1:46175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 16:35:48.770407 25532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:48.769787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:46175 (local address 127.0.0.1:39146)
0926 16:35:48.770044 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:48.770157 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.1:46175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260926 16:35:48.771559 25532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:48.771185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:46175 (local address 127.0.0.1:39154)
0926 16:35:48.771349 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:48.771434 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.1:46175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 16:35:48.811900 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 25259
I20260926 16:35:48.835361 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 25389
I20260926 16:35:48.843895 25529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:48.836299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48462 (local address 127.6.33.3:39593)
0926 16:35:48.836502 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:48.836509 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:48.836530 (+ 21us) server_negotiation.cc:407] Connection header received
0926 16:35:48.836597 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:48.836602 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:48.836705 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:48.836849 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:35:48.837313 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:48.838571 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:48.839976 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:48.840661 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:48.840826 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:48.842060 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:35:48.842074 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:35:48.842105 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:35:48.842992 (+ 887us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:35:48.843066 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:35:48.843097 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:35:48.843251 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:48.843378 (+ 127us) server_negotiation.cc:299] Negotiation successful
0926 16:35:48.843495 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 16:35:48.844847 25511 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:35:48.852802 25532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:48.852305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:46175 (local address 127.0.0.1:39156)
0926 16:35:48.852505 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:48.852596 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.1:46175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260926 16:35:48.853873 25532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:48.853474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:46175 (local address 127.0.0.1:39160)
0926 16:35:48.853682 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:48.853766 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.1:46175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260926 16:35:48.966254 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 25389
I20260926 16:35:48.985904 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 25038
I20260926 16:35:48.987092 25099 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:35:49.100342 25532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:49.099593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:46175 (local address 127.0.0.1:39174)
0926 16:35:49.099824 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:49.099949 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.1:46175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260926 16:35:49.100970 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 25038
2026-09-26T16:35:49Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2873 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:35:49 dist-test-slave-khd3 krb5kdc[25568](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:35:49 dist-test-slave-khd3 krb5kdc[25568](info): set up 2 sockets
Sep 26 16:35:49 dist-test-slave-khd3 krb5kdc[25568](info): commencing operation
krb5kdc: starting...
W20260926 16:35:50.290030 6276 mini_kdc.cc:121] Time spent starting KDC: real 1.146s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Sep 26 16:35:50 dist-test-slave-khd3 krb5kdc[25568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:35:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:35:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab.
Entry for principal kudu/127.6.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab.
I20260926 16:35:50.544147 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:36817
--webserver_interface=127.6.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:44567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.33.62:36817
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:35:50.879524 25581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:35:50.879848 25581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:35:50.879943 25581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:35:50.887305 25581 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:35:50.887432 25581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:35:50.887518 25581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:35:50.887609 25581 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:35:50.887681 25581 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:35:50.898065 25581 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:44567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.33.62:36817
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:36817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:35:50.899501 25581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:35:50.900996 25581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:35:50.914825 25587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:35:50.914885 25586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:35:50.915443 25589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:35:50.915793 25581 server_base.cc:1034] running on GCE node
I20260926 16:35:50.916698 25581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:35:50.918341 25581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:35:50.919612 25581 hybrid_clock.cc:648] HybridClock initialized: now 1790440550919561 us; error 60 us; skew 500 ppm
I20260926 16:35:50.919998 25581 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:35:50 dist-test-slave-khd3 krb5kdc[25568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440550, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:35:50.923308 25581 init.cc:377] Logged in from keytab as kudu/127.6.33.62@KRBTEST.COM (short username kudu)
I20260926 16:35:50.925081 25581 webserver.cc:460] Webserver started at http://127.6.33.62:34317/ using document root <none> and password file <none>
I20260926 16:35:50.925801 25581 fs_manager.cc:359] Metadata directory not provided
I20260926 16:35:50.925946 25581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:35:50.926267 25581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:35:50.929327 25581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/master-0/data/instance:
uuid: "896c4f0c5de6461fa12759890e9886a3"
format_stamp: "Formatted at 2026-09-26 16:35:50 on dist-test-slave-khd3"
server_key: "55970650840bb39d2b1084fca84d8a8c"
server_key_iv: "199502dcbdc0e345adc04f30c817ba63"
server_key_version: "encryptionkey@0"
I20260926 16:35:50.930079 25581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/master-0/wal/instance:
uuid: "896c4f0c5de6461fa12759890e9886a3"
format_stamp: "Formatted at 2026-09-26 16:35:50 on dist-test-slave-khd3"
server_key: "55970650840bb39d2b1084fca84d8a8c"
server_key_iv: "199502dcbdc0e345adc04f30c817ba63"
server_key_version: "encryptionkey@0"
I20260926 16:35:50.935487 25581 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 16:35:50.939327 25596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:35:50.941022 25581 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 16:35:50.941215 25581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/master-0/wal
uuid: "896c4f0c5de6461fa12759890e9886a3"
format_stamp: "Formatted at 2026-09-26 16:35:50 on dist-test-slave-khd3"
server_key: "55970650840bb39d2b1084fca84d8a8c"
server_key_iv: "199502dcbdc0e345adc04f30c817ba63"
server_key_version: "encryptionkey@0"
I20260926 16:35:50.941424 25581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:35:50.964406 25581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:35:50.969070 25581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:35:50.969415 25581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:35:50.989806 25581 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.62:36817
I20260926 16:35:50.989838 25647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.62:36817 every 8 connection(s)
I20260926 16:35:50.991300 25581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
I20260926 16:35:50.995342 25648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:35:51.000188 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 25581
I20260926 16:35:51.000403 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/master-0/wal/instance
I20260926 16:35:51.001017 6276 external_mini_cluster.cc:1428] Setting key 7fbd2c7aae2199b7013aaed68267a0a6
I20260926 16:35:51.009936 25648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 896c4f0c5de6461fa12759890e9886a3: Bootstrap starting.
I20260926 16:35:51.013795 25648 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 896c4f0c5de6461fa12759890e9886a3: Neither blocks nor log segments found. Creating new log.
I20260926 16:35:51.014937 25648 log.cc:824] T 00000000000000000000000000000000 P 896c4f0c5de6461fa12759890e9886a3: Log is configured to *not* fsync() on all Append() calls
I20260926 16:35:51.018448 25648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 896c4f0c5de6461fa12759890e9886a3: No bootstrap required, opened a new log
Sep 26 16:35:51 dist-test-slave-khd3 krb5kdc[25568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:35:51.027426 25648 raft_consensus.cc:348] T 00000000000000000000000000000000 P 896c4f0c5de6461fa12759890e9886a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "896c4f0c5de6461fa12759890e9886a3" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 36817 } }
I20260926 16:35:51.027881 25648 raft_consensus.cc:374] T 00000000000000000000000000000000 P 896c4f0c5de6461fa12759890e9886a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:35:51.028047 25648 raft_consensus.cc:729] T 00000000000000000000000000000000 P 896c4f0c5de6461fa12759890e9886a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 896c4f0c5de6461fa12759890e9886a3, State: Initialized, Role: FOLLOWER
I20260926 16:35:51.028671 25648 consensus_queue.cc:260] T 00000000000000000000000000000000 P 896c4f0c5de6461fa12759890e9886a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "896c4f0c5de6461fa12759890e9886a3" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 36817 } }
I20260926 16:35:51.029028 25648 raft_consensus.cc:388] T 00000000000000000000000000000000 P 896c4f0c5de6461fa12759890e9886a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:35:51.029181 25648 raft_consensus.cc:482] T 00000000000000000000000000000000 P 896c4f0c5de6461fa12759890e9886a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:35:51.029383 25648 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 896c4f0c5de6461fa12759890e9886a3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:35:51.031549 25651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:51.003209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43690 (local address 127.6.33.62:36817)
0926 16:35:51.004066 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:51.004080 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:51.004128 (+ 48us) server_negotiation.cc:407] Connection header received
0926 16:35:51.005918 (+ 1790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:51.005944 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:51.006408 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:51.006863 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:35:51.008441 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:51.009917 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:51.011310 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:51.012026 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:51.012243 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:51.022799 (+ 10556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:35:51.022809 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:35:51.022826 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:35:51.022882 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:35:51.026408 (+ 3526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:35:51.027231 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:35:51.027238 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:35:51.027245 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:35:51.027421 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:35:51.027822 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:35:51.027830 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:35:51.027836 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:35:51.028677 (+ 841us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:35:51.029023 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:51.029533 (+ 510us) server_negotiation.cc:299] Negotiation successful
0926 16:35:51.030029 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":415,"threads_started":1}
I20260926 16:35:51.032809 25648 raft_consensus.cc:504] T 00000000000000000000000000000000 P 896c4f0c5de6461fa12759890e9886a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "896c4f0c5de6461fa12759890e9886a3" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 36817 } }
I20260926 16:35:51.033349 25648 leader_election.cc:302] T 00000000000000000000000000000000 P 896c4f0c5de6461fa12759890e9886a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 896c4f0c5de6461fa12759890e9886a3; no voters:
I20260926 16:35:51.033972 25648 leader_election.cc:288] T 00000000000000000000000000000000 P 896c4f0c5de6461fa12759890e9886a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:35:51.034114 25653 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 896c4f0c5de6461fa12759890e9886a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:35:51.035455 25653 raft_consensus.cc:686] T 00000000000000000000000000000000 P 896c4f0c5de6461fa12759890e9886a3 [term 1 LEADER]: Becoming Leader. State: Replica: 896c4f0c5de6461fa12759890e9886a3, State: Running, Role: LEADER
I20260926 16:35:51.036007 25653 consensus_queue.cc:237] T 00000000000000000000000000000000 P 896c4f0c5de6461fa12759890e9886a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "896c4f0c5de6461fa12759890e9886a3" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 36817 } }
I20260926 16:35:51.036310 25648 sys_catalog.cc:564] T 00000000000000000000000000000000 P 896c4f0c5de6461fa12759890e9886a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:35:51.038772 25655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 896c4f0c5de6461fa12759890e9886a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 896c4f0c5de6461fa12759890e9886a3. Latest consensus state: current_term: 1 leader_uuid: "896c4f0c5de6461fa12759890e9886a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "896c4f0c5de6461fa12759890e9886a3" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 36817 } } }
I20260926 16:35:51.039110 25655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 896c4f0c5de6461fa12759890e9886a3 [sys.catalog]: This master's current role is: LEADER
I20260926 16:35:51.039856 25654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 896c4f0c5de6461fa12759890e9886a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "896c4f0c5de6461fa12759890e9886a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "896c4f0c5de6461fa12759890e9886a3" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 36817 } } }
I20260926 16:35:51.040163 25654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 896c4f0c5de6461fa12759890e9886a3 [sys.catalog]: This master's current role is: LEADER
I20260926 16:35:51.040272 25660 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:35:51.049865 25660 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:35:51.059044 25660 catalog_manager.cc:1347] Generated new cluster ID: 705b3d41029f477ebfae5b4eb0041a2a
I20260926 16:35:51.059197 25660 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:35:51.068094 25660 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:35:51.069615 25660 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:35:51.080451 25660 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 896c4f0c5de6461fa12759890e9886a3: Generated new TSK 0
I20260926 16:35:51.081691 25660 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.33.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab.
Entry for principal kudu/127.6.33.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab.
I20260926 16:35:51.143967 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.1:0
--local_ip_for_outbound_sockets=127.6.33.1
--webserver_interface=127.6.33.1
--webserver_port=0
--tserver_master_addrs=127.6.33.62:36817
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab
--principal=kudu/127.6.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:44567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:35:51.484447 25674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:35:51.484732 25674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:35:51.484918 25674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:35:51.492113 25674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:35:51.492233 25674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:35:51.492349 25674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.1
I20260926 16:35:51.502421 25674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:44567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.1.keytab
--principal=kudu/127.6.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.33.1
--webserver_port=0
--tserver_master_addrs=127.6.33.62:36817
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.1
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:35:51.503747 25674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:35:51.505132 25674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:35:51.519297 25680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:35:51.519363 25679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:35:51.519927 25682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:35:51.522370 25674 server_base.cc:1034] running on GCE node
I20260926 16:35:51.523042 25674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:35:51.524108 25674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:35:51.525338 25674 hybrid_clock.cc:648] HybridClock initialized: now 1790440551525271 us; error 66 us; skew 500 ppm
I20260926 16:35:51.525673 25674 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:35:51 dist-test-slave-khd3 krb5kdc[25568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440551, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:35:51.528892 25674 init.cc:377] Logged in from keytab as kudu/127.6.33.1@KRBTEST.COM (short username kudu)
I20260926 16:35:51.530763 25674 webserver.cc:460] Webserver started at http://127.6.33.1:46419/ using document root <none> and password file <none>
I20260926 16:35:51.531533 25674 fs_manager.cc:359] Metadata directory not provided
I20260926 16:35:51.531687 25674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:35:51.532012 25674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:35:51.535126 25674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/ts-0/data/instance:
uuid: "352297845b9640cd915bb38ffc2c2d32"
format_stamp: "Formatted at 2026-09-26 16:35:51 on dist-test-slave-khd3"
server_key: "a93519a22f7bec1aa96f147bd7ffc6f2"
server_key_iv: "7dee7b9ecbeb30f6b24509a6b7795cbe"
server_key_version: "encryptionkey@0"
I20260926 16:35:51.535959 25674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/ts-0/wal/instance:
uuid: "352297845b9640cd915bb38ffc2c2d32"
format_stamp: "Formatted at 2026-09-26 16:35:51 on dist-test-slave-khd3"
server_key: "a93519a22f7bec1aa96f147bd7ffc6f2"
server_key_iv: "7dee7b9ecbeb30f6b24509a6b7795cbe"
server_key_version: "encryptionkey@0"
I20260926 16:35:51.541895 25674 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 16:35:51.545806 25689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:35:51.547344 25674 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:35:51.547531 25674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/ts-0/wal
uuid: "352297845b9640cd915bb38ffc2c2d32"
format_stamp: "Formatted at 2026-09-26 16:35:51 on dist-test-slave-khd3"
server_key: "a93519a22f7bec1aa96f147bd7ffc6f2"
server_key_iv: "7dee7b9ecbeb30f6b24509a6b7795cbe"
server_key_version: "encryptionkey@0"
I20260926 16:35:51.547740 25674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:35:51.568410 25674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:35:51.573166 25674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:35:51.573514 25674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:35:51.574545 25674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:35:51.575918 25674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:35:51.576030 25674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:35:51.576180 25674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:35:51.576253 25674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:35:51.611199 25674 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.1:36239
I20260926 16:35:51.611253 25801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.1:36239 every 8 connection(s)
I20260926 16:35:51.612594 25674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
I20260926 16:35:51.619617 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 25674
I20260926 16:35:51.619783 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/ts-0/wal/instance
I20260926 16:35:51.620239 6276 external_mini_cluster.cc:1428] Setting key 831f33880551c63083453e51fdd5ecd8
Sep 26 16:35:51 dist-test-slave-khd3 krb5kdc[25568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440551, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.1@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:35:51.636737 25804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:51.615197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:49933 (local address 127.6.33.62:36817)
0926 16:35:51.615643 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:51.615649 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:51.616693 (+ 1044us) server_negotiation.cc:407] Connection header received
0926 16:35:51.618631 (+ 1938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:51.618635 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:51.618702 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:51.618790 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:35:51.620328 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:51.621216 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:51.623497 (+ 2281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:51.624259 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:51.624409 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:51.629448 (+ 5039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:35:51.629454 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:35:51.629459 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:35:51.629504 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:35:51.633213 (+ 3709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:35:51.634027 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:35:51.634036 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:35:51.634042 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:35:51.634193 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:35:51.634614 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:35:51.634621 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:35:51.634628 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:35:51.635017 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:35:51.635179 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:51.635933 (+ 754us) server_negotiation.cc:299] Negotiation successful
0926 16:35:51.636200 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":199,"threads_started":1}
I20260926 16:35:51.637540 25805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:51.615771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36817 (local address 127.6.33.1:49933)
0926 16:35:51.616528 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:51.616576 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 16:35:51.618354 (+ 1778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:35:51.618995 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:35:51.619002 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:35:51.619521 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:35:51.620161 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:51.620178 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:51.621398 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:51.621402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:51.623318 (+ 1916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:51.623336 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:51.624729 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:51.624738 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:51.625109 (+ 371us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:51.625964 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:35:51.626004 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:35:51.629180 (+ 3176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:35:51.633402 (+ 4222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:35:51.633413 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:35:51.633428 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:35:51.633891 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:35:51.634305 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:35:51.634308 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:35:51.634310 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:35:51.634510 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:35:51.635181 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:35:51.635187 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:35:51.635529 (+ 342us) client_negotiation.cc:769] Sending connection context
0926 16:35:51.635788 (+ 259us) client_negotiation.cc:240] Negotiation successful
0926 16:35:51.636225 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":314,"threads_started":1}
I20260926 16:35:51.639365 25802 heartbeater.cc:347] Connected to a master server at 127.6.33.62:36817
I20260926 16:35:51.639699 25802 heartbeater.cc:464] Registering TS with master...
I20260926 16:35:51.640637 25802 heartbeater.cc:511] Master 127.6.33.62:36817 requested a full tablet report, sending...
I20260926 16:35:51.643285 25613 ts_manager.cc:194] Registered new tserver with Master: 352297845b9640cd915bb38ffc2c2d32 (127.6.33.1:36239)
I20260926 16:35:51.645552 25613 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.33.1@KRBTEST.COM'} at 127.6.33.1:49933
WARNING: no policy specified for kudu/127.6.33.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab.
Entry for principal kudu/127.6.33.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab.
I20260926 16:35:51.681921 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.2:0
--local_ip_for_outbound_sockets=127.6.33.2
--webserver_interface=127.6.33.2
--webserver_port=0
--tserver_master_addrs=127.6.33.62:36817
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab
--principal=kudu/127.6.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:44567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:35:52.022339 25808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:35:52.022747 25808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:35:52.022957 25808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:35:52.030797 25808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:35:52.030915 25808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:35:52.031030 25808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.2
I20260926 16:35:52.041479 25808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:44567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.2.keytab
--principal=kudu/127.6.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.33.2
--webserver_port=0
--tserver_master_addrs=127.6.33.62:36817
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.2
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:35:52.042915 25808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:35:52.044263 25808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:35:52.058473 25813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:35:52.058506 25814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:35:52.059172 25816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:35:52.059500 25808 server_base.cc:1034] running on GCE node
I20260926 16:35:52.060163 25808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:35:52.061304 25808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:35:52.062559 25808 hybrid_clock.cc:648] HybridClock initialized: now 1790440552062499 us; error 50 us; skew 500 ppm
I20260926 16:35:52.062974 25808 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:35:52 dist-test-slave-khd3 krb5kdc[25568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440552, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:35:52.066375 25808 init.cc:377] Logged in from keytab as kudu/127.6.33.2@KRBTEST.COM (short username kudu)
I20260926 16:35:52.067973 25808 webserver.cc:460] Webserver started at http://127.6.33.2:37621/ using document root <none> and password file <none>
I20260926 16:35:52.068692 25808 fs_manager.cc:359] Metadata directory not provided
I20260926 16:35:52.068864 25808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:35:52.069197 25808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:35:52.072162 25808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/ts-1/data/instance:
uuid: "3fec1956565c4b87b1fe9937797146b6"
format_stamp: "Formatted at 2026-09-26 16:35:52 on dist-test-slave-khd3"
server_key: "b38891ba598e199bf66a81600a2d55a7"
server_key_iv: "854afdbce0e269e060e375c0bbc117a1"
server_key_version: "encryptionkey@0"
I20260926 16:35:52.073040 25808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/ts-1/wal/instance:
uuid: "3fec1956565c4b87b1fe9937797146b6"
format_stamp: "Formatted at 2026-09-26 16:35:52 on dist-test-slave-khd3"
server_key: "b38891ba598e199bf66a81600a2d55a7"
server_key_iv: "854afdbce0e269e060e375c0bbc117a1"
server_key_version: "encryptionkey@0"
I20260926 16:35:52.078377 25808 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 16:35:52.081909 25823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:35:52.083303 25808 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260926 16:35:52.083473 25808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/ts-1/wal
uuid: "3fec1956565c4b87b1fe9937797146b6"
format_stamp: "Formatted at 2026-09-26 16:35:52 on dist-test-slave-khd3"
server_key: "b38891ba598e199bf66a81600a2d55a7"
server_key_iv: "854afdbce0e269e060e375c0bbc117a1"
server_key_version: "encryptionkey@0"
I20260926 16:35:52.083676 25808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:35:52.099120 25808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:35:52.103830 25808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:35:52.104157 25808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:35:52.105280 25808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:35:52.106984 25808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:35:52.107091 25808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:35:52.107256 25808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:35:52.107323 25808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:35:52.141940 25808 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.2:46669
I20260926 16:35:52.141958 25935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.2:46669 every 8 connection(s)
I20260926 16:35:52.143339 25808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
I20260926 16:35:52.147570 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 25808
I20260926 16:35:52.147778 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/ts-1/wal/instance
I20260926 16:35:52.148401 6276 external_mini_cluster.cc:1428] Setting key 99a2bb9073a433b1dc40ab4a20077f8d
Sep 26 16:35:52 dist-test-slave-khd3 krb5kdc[25568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440552, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.2@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:35:52.165719 25938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:52.146266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:55645 (local address 127.6.33.62:36817)
0926 16:35:52.146730 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:52.146734 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:52.147824 (+ 1090us) server_negotiation.cc:407] Connection header received
0926 16:35:52.150252 (+ 2428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:52.150260 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:52.150349 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:52.150441 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:35:52.152124 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:52.153375 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:52.154944 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:52.155710 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:52.155860 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:52.159701 (+ 3841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:35:52.159706 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:35:52.159709 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:35:52.159745 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:35:52.162472 (+ 2727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:35:52.163263 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:35:52.163269 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:35:52.163272 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:35:52.163415 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:35:52.163863 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:35:52.163868 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:35:52.163871 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:35:52.164145 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:35:52.164299 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:52.165064 (+ 765us) server_negotiation.cc:299] Negotiation successful
0926 16:35:52.165298 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":253,"threads_started":1}
I20260926 16:35:52.166685 25939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:52.146780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36817 (local address 127.6.33.2:55645)
0926 16:35:52.147604 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:52.147688 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0926 16:35:52.149938 (+ 2250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:35:52.150719 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:35:52.150727 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:35:52.151234 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:35:52.151892 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:52.151935 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:52.153512 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:52.153516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:52.154795 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:52.154807 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:52.155932 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:52.155937 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:52.156157 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:52.156915 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:35:52.156950 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:35:52.159448 (+ 2498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:35:52.162639 (+ 3191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:35:52.162650 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:35:52.162670 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:35:52.163137 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:35:52.163534 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:35:52.163540 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:35:52.163543 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:35:52.163747 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:35:52.164271 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:35:52.164279 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:35:52.164646 (+ 367us) client_negotiation.cc:769] Sending connection context
0926 16:35:52.164973 (+ 327us) client_negotiation.cc:240] Negotiation successful
0926 16:35:52.165547 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":301,"threads_started":1}
I20260926 16:35:52.168202 25936 heartbeater.cc:347] Connected to a master server at 127.6.33.62:36817
I20260926 16:35:52.168540 25936 heartbeater.cc:464] Registering TS with master...
I20260926 16:35:52.169332 25936 heartbeater.cc:511] Master 127.6.33.62:36817 requested a full tablet report, sending...
I20260926 16:35:52.170943 25613 ts_manager.cc:194] Registered new tserver with Master: 3fec1956565c4b87b1fe9937797146b6 (127.6.33.2:46669)
I20260926 16:35:52.171815 25613 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.33.2@KRBTEST.COM'} at 127.6.33.2:55645
WARNING: no policy specified for kudu/127.6.33.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab.
Entry for principal kudu/127.6.33.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab.
I20260926 16:35:52.208106 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.3:0
--local_ip_for_outbound_sockets=127.6.33.3
--webserver_interface=127.6.33.3
--webserver_port=0
--tserver_master_addrs=127.6.33.62:36817
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab
--principal=kudu/127.6.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:44567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:35:52.541992 25942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:35:52.542343 25942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:35:52.542517 25942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:35:52.550390 25942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:35:52.550563 25942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:35:52.550783 25942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.3
I20260926 16:35:52.561113 25942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:44567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.3.keytab
--principal=kudu/127.6.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.33.3
--webserver_port=0
--tserver_master_addrs=127.6.33.62:36817
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.3
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:35:52.562388 25942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:35:52.563869 25942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:35:52.577232 25947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:35:52.577301 25950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:35:52.577324 25948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:35:52.578075 25942 server_base.cc:1034] running on GCE node
I20260926 16:35:52.578689 25942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:35:52.579699 25942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:35:52.580914 25942 hybrid_clock.cc:648] HybridClock initialized: now 1790440552580856 us; error 52 us; skew 500 ppm
I20260926 16:35:52.581441 25942 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:35:52 dist-test-slave-khd3 krb5kdc[25568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440552, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:35:52.584599 25942 init.cc:377] Logged in from keytab as kudu/127.6.33.3@KRBTEST.COM (short username kudu)
I20260926 16:35:52.586135 25942 webserver.cc:460] Webserver started at http://127.6.33.3:42175/ using document root <none> and password file <none>
I20260926 16:35:52.586833 25942 fs_manager.cc:359] Metadata directory not provided
I20260926 16:35:52.586988 25942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:35:52.587297 25942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:35:52.590415 25942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/ts-2/data/instance:
uuid: "bac98ca197074cf881f307af40955cf1"
format_stamp: "Formatted at 2026-09-26 16:35:52 on dist-test-slave-khd3"
server_key: "3c197f495ba73a7425a3c3a57b37a483"
server_key_iv: "6b8a7623aa4f66cbea34ae2a482ff1db"
server_key_version: "encryptionkey@0"
I20260926 16:35:52.591243 25942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/ts-2/wal/instance:
uuid: "bac98ca197074cf881f307af40955cf1"
format_stamp: "Formatted at 2026-09-26 16:35:52 on dist-test-slave-khd3"
server_key: "3c197f495ba73a7425a3c3a57b37a483"
server_key_iv: "6b8a7623aa4f66cbea34ae2a482ff1db"
server_key_version: "encryptionkey@0"
I20260926 16:35:52.596671 25942 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 16:35:52.600234 25957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:35:52.601792 25942 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 16:35:52.601977 25942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/ts-2/wal
uuid: "bac98ca197074cf881f307af40955cf1"
format_stamp: "Formatted at 2026-09-26 16:35:52 on dist-test-slave-khd3"
server_key: "3c197f495ba73a7425a3c3a57b37a483"
server_key_iv: "6b8a7623aa4f66cbea34ae2a482ff1db"
server_key_version: "encryptionkey@0"
I20260926 16:35:52.602206 25942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:35:52.623957 25942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:35:52.628049 25942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:35:52.628342 25942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:35:52.629406 25942 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:35:52.630748 25942 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:35:52.630851 25942 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:35:52.630995 25942 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:35:52.631064 25942 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:35:52.649657 25802 heartbeater.cc:503] Master 127.6.33.62:36817 was elected leader, sending a full tablet report...
I20260926 16:35:52.665518 25942 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.3:46537
I20260926 16:35:52.665551 26069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.3:46537 every 8 connection(s)
I20260926 16:35:52.666954 25942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
I20260926 16:35:52.673323 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 25942
I20260926 16:35:52.673516 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/ts-2/wal/instance
I20260926 16:35:52.673952 6276 external_mini_cluster.cc:1428] Setting key 16335563718d105e0f89e98f511d8ea9
Sep 26 16:35:52 dist-test-slave-khd3 krb5kdc[25568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440552, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.3@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:35:52.687991 26072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:52.669702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.3:57443 (local address 127.6.33.62:36817)
0926 16:35:52.670165 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:52.670169 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:52.671181 (+ 1012us) server_negotiation.cc:407] Connection header received
0926 16:35:52.673088 (+ 1907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:52.673094 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:52.673194 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:52.673331 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:35:52.675221 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:52.676331 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:52.678090 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:52.678471 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:52.678568 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:52.681967 (+ 3399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:35:52.681971 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:35:52.681975 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:35:52.682007 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:35:52.684574 (+ 2567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:35:52.685427 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:35:52.685432 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:35:52.685435 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:35:52.685602 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:35:52.686068 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:35:52.686071 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:35:52.686073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:35:52.686346 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:35:52.686482 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:52.687287 (+ 805us) server_negotiation.cc:299] Negotiation successful
0926 16:35:52.687541 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":214,"threads_started":1}
I20260926 16:35:52.689419 26073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:52.670274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:36817 (local address 127.6.33.3:57443)
0926 16:35:52.671008 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:52.671056 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 16:35:52.672729 (+ 1673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:35:52.673612 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:35:52.673647 (+ 35us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:35:52.674250 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:35:52.674967 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:52.674988 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:52.676486 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:52.676490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:52.677958 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:52.677971 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:52.678686 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:52.678691 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:52.678887 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:52.679515 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:35:52.679542 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:35:52.681770 (+ 2228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:35:52.684714 (+ 2944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:35:52.684720 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:35:52.684733 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:35:52.685241 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:35:52.685746 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:35:52.685750 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:35:52.685753 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:35:52.685952 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:35:52.686476 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:35:52.686483 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:35:52.686852 (+ 369us) client_negotiation.cc:769] Sending connection context
0926 16:35:52.687167 (+ 315us) client_negotiation.cc:240] Negotiation successful
0926 16:35:52.687765 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":276,"threads_started":1}
I20260926 16:35:52.690937 26070 heartbeater.cc:347] Connected to a master server at 127.6.33.62:36817
I20260926 16:35:52.691303 26070 heartbeater.cc:464] Registering TS with master...
I20260926 16:35:52.692075 26070 heartbeater.cc:511] Master 127.6.33.62:36817 requested a full tablet report, sending...
I20260926 16:35:52.693636 25613 ts_manager.cc:194] Registered new tserver with Master: bac98ca197074cf881f307af40955cf1 (127.6.33.3:46537)
I20260926 16:35:52.694515 25613 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.33.3@KRBTEST.COM'} at 127.6.33.3:57443
I20260926 16:35:52.702201 6276 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:35:52.719631 26072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:52.706793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43702 (local address 127.6.33.62:36817)
0926 16:35:52.707104 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:52.707111 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:52.707445 (+ 334us) server_negotiation.cc:407] Connection header received
0926 16:35:52.707607 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:52.707611 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:52.707681 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:52.707770 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:35:52.709176 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:52.709908 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:52.711315 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:52.711743 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:52.711844 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:52.714407 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:35:52.714411 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:35:52.714415 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:35:52.714447 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:35:52.717189 (+ 2742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:35:52.717832 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:35:52.717836 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:35:52.717838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:35:52.717947 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:35:52.718242 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:35:52.718245 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:35:52.718247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:35:52.718584 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:35:52.718764 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:52.719062 (+ 298us) server_negotiation.cc:299] Negotiation successful
0926 16:35:52.719234 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 16:35:52.723506 25613 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43702:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:35:52.730415 25613 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:35:52.762125 26083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:52.742737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.3:46537 (local address 127.0.0.1:46214)
0926 16:35:52.743674 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:52.743716 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 16:35:52.743887 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:35:52.756113 (+ 12226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:35:52.756120 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:35:52.756152 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:35:52.756485 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:52.756496 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:52.758063 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:52.758068 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:52.759589 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:52.759602 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:52.761058 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:52.761065 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:52.761281 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:52.761353 (+ 72us) client_negotiation.cc:769] Sending connection context
0926 16:35:52.761507 (+ 154us) client_negotiation.cc:240] Negotiation successful
0926 16:35:52.761722 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":707,"thread_start_us":560,"threads_started":1}
I20260926 16:35:52.763020 26084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:52.743097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46214 (local address 127.6.33.3:46537)
0926 16:35:52.755033 (+ 11936us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:52.755040 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:52.755070 (+ 30us) server_negotiation.cc:407] Connection header received
0926 16:35:52.755154 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:52.755160 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:52.755453 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:52.755768 (+ 315us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:35:52.756661 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:52.757920 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:52.759751 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:52.760862 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:52.760988 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:52.761241 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:52.761914 (+ 673us) server_negotiation.cc:299] Negotiation successful
0926 16:35:52.762094 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11766,"thread_start_us":11592,"threads_started":1}
I20260926 16:35:52.766887 26005 tablet_service.cc:1467] Processing CreateTablet for tablet 34d072cff51f4900ad7185b674e86d05 (DEFAULT_TABLE table=test-table [id=88832550bb464254add3c79f5a697a17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:35:52.768497 26005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34d072cff51f4900ad7185b674e86d05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:35:52.769500 26088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:52.745129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50880 (local address 127.6.33.1:36239)
0926 16:35:52.757161 (+ 12032us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:52.757172 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:52.762216 (+ 5044us) server_negotiation.cc:407] Connection header received
0926 16:35:52.762325 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:52.762332 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:52.762586 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:52.762786 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:35:52.763605 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:52.764610 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:52.767238 (+ 2628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:52.768422 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:52.768569 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:52.768818 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:52.769025 (+ 207us) server_negotiation.cc:299] Negotiation successful
0926 16:35:52.769187 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11848,"thread_start_us":11672,"threads_started":1}
I20260926 16:35:52.769531 26086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:52.744122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:36239 (local address 127.0.0.1:50880)
0926 16:35:52.761912 (+ 17790us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:52.761937 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 16:35:52.762057 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:35:52.763071 (+ 1014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:35:52.763077 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:35:52.763127 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:35:52.763444 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:52.763453 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:52.764745 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:52.764784 (+ 39us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:52.766717 (+ 1933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:52.766735 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:52.768628 (+ 1893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:52.768635 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:52.768833 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:52.768877 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 16:35:52.768979 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 16:35:52.769109 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17668,"mutex_wait_us":1,"thread_start_us":17537,"threads_started":1}
I20260926 16:35:52.770292 26085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:52.743311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.2:46669 (local address 127.0.0.1:54520)
0926 16:35:52.753964 (+ 10653us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:52.753998 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 16:35:52.754175 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:35:52.762379 (+ 8204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:35:52.762384 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:35:52.762411 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:35:52.762745 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:52.762752 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:52.765670 (+ 2918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:52.765675 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:52.767686 (+ 2011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:52.767701 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:52.769564 (+ 1863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:52.769570 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:52.769742 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:52.769774 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 16:35:52.769840 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 16:35:52.769965 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10454,"thread_start_us":10972,"threads_started":1}
I20260926 16:35:52.770906 26087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:52.744458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54520 (local address 127.6.33.2:46669)
0926 16:35:52.761710 (+ 17252us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:52.761719 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:52.761748 (+ 29us) server_negotiation.cc:407] Connection header received
0926 16:35:52.761847 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:52.761858 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:52.762135 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:52.762903 (+ 768us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:35:52.763079 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:52.765522 (+ 2443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:52.767850 (+ 2328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:52.769317 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:52.770099 (+ 782us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:52.770278 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:52.770339 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 16:35:52.770601 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17091,"thread_start_us":16941,"threads_started":1}
I20260926 16:35:52.773766 25737 tablet_service.cc:1467] Processing CreateTablet for tablet 34d072cff51f4900ad7185b674e86d05 (DEFAULT_TABLE table=test-table [id=88832550bb464254add3c79f5a697a17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:35:52.773859 25871 tablet_service.cc:1467] Processing CreateTablet for tablet 34d072cff51f4900ad7185b674e86d05 (DEFAULT_TABLE table=test-table [id=88832550bb464254add3c79f5a697a17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:35:52.775236 25871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34d072cff51f4900ad7185b674e86d05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:35:52.775295 25737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34d072cff51f4900ad7185b674e86d05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:35:52.784494 26089 tablet_bootstrap.cc:492] T 34d072cff51f4900ad7185b674e86d05 P bac98ca197074cf881f307af40955cf1: Bootstrap starting.
I20260926 16:35:52.786875 26090 tablet_bootstrap.cc:492] T 34d072cff51f4900ad7185b674e86d05 P 3fec1956565c4b87b1fe9937797146b6: Bootstrap starting.
I20260926 16:35:52.788447 26089 tablet_bootstrap.cc:654] T 34d072cff51f4900ad7185b674e86d05 P bac98ca197074cf881f307af40955cf1: Neither blocks nor log segments found. Creating new log.
I20260926 16:35:52.789721 26089 log.cc:824] T 34d072cff51f4900ad7185b674e86d05 P bac98ca197074cf881f307af40955cf1: Log is configured to *not* fsync() on all Append() calls
I20260926 16:35:52.789985 26091 tablet_bootstrap.cc:492] T 34d072cff51f4900ad7185b674e86d05 P 352297845b9640cd915bb38ffc2c2d32: Bootstrap starting.
I20260926 16:35:52.790618 26090 tablet_bootstrap.cc:654] T 34d072cff51f4900ad7185b674e86d05 P 3fec1956565c4b87b1fe9937797146b6: Neither blocks nor log segments found. Creating new log.
I20260926 16:35:52.791652 26090 log.cc:824] T 34d072cff51f4900ad7185b674e86d05 P 3fec1956565c4b87b1fe9937797146b6: Log is configured to *not* fsync() on all Append() calls
I20260926 16:35:52.794037 26091 tablet_bootstrap.cc:654] T 34d072cff51f4900ad7185b674e86d05 P 352297845b9640cd915bb38ffc2c2d32: Neither blocks nor log segments found. Creating new log.
I20260926 16:35:52.794884 26089 tablet_bootstrap.cc:492] T 34d072cff51f4900ad7185b674e86d05 P bac98ca197074cf881f307af40955cf1: No bootstrap required, opened a new log
I20260926 16:35:52.794936 26090 tablet_bootstrap.cc:492] T 34d072cff51f4900ad7185b674e86d05 P 3fec1956565c4b87b1fe9937797146b6: No bootstrap required, opened a new log
I20260926 16:35:52.795177 26089 ts_tablet_manager.cc:1397] T 34d072cff51f4900ad7185b674e86d05 P bac98ca197074cf881f307af40955cf1: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260926 16:35:52.795248 26090 ts_tablet_manager.cc:1397] T 34d072cff51f4900ad7185b674e86d05 P 3fec1956565c4b87b1fe9937797146b6: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.008s
I20260926 16:35:52.795311 26091 log.cc:824] T 34d072cff51f4900ad7185b674e86d05 P 352297845b9640cd915bb38ffc2c2d32: Log is configured to *not* fsync() on all Append() calls
I20260926 16:35:52.798403 26091 tablet_bootstrap.cc:492] T 34d072cff51f4900ad7185b674e86d05 P 352297845b9640cd915bb38ffc2c2d32: No bootstrap required, opened a new log
I20260926 16:35:52.798722 26091 ts_tablet_manager.cc:1397] T 34d072cff51f4900ad7185b674e86d05 P 352297845b9640cd915bb38ffc2c2d32: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.002s
I20260926 16:35:52.802989 26089 raft_consensus.cc:348] T 34d072cff51f4900ad7185b674e86d05 P bac98ca197074cf881f307af40955cf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac98ca197074cf881f307af40955cf1" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 46537 } } peers { permanent_uuid: "352297845b9640cd915bb38ffc2c2d32" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 36239 } } peers { permanent_uuid: "3fec1956565c4b87b1fe9937797146b6" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 46669 } }
I20260926 16:35:52.803357 26089 raft_consensus.cc:374] T 34d072cff51f4900ad7185b674e86d05 P bac98ca197074cf881f307af40955cf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:35:52.803457 26089 raft_consensus.cc:729] T 34d072cff51f4900ad7185b674e86d05 P bac98ca197074cf881f307af40955cf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bac98ca197074cf881f307af40955cf1, State: Initialized, Role: FOLLOWER
I20260926 16:35:52.804153 26089 consensus_queue.cc:260] T 34d072cff51f4900ad7185b674e86d05 P bac98ca197074cf881f307af40955cf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac98ca197074cf881f307af40955cf1" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 46537 } } peers { permanent_uuid: "352297845b9640cd915bb38ffc2c2d32" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 36239 } } peers { permanent_uuid: "3fec1956565c4b87b1fe9937797146b6" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 46669 } }
I20260926 16:35:52.805404 26070 heartbeater.cc:503] Master 127.6.33.62:36817 was elected leader, sending a full tablet report...
I20260926 16:35:52.806161 26089 ts_tablet_manager.cc:1428] T 34d072cff51f4900ad7185b674e86d05 P bac98ca197074cf881f307af40955cf1: Time spent starting tablet: real 0.011s user 0.008s sys 0.005s
I20260926 16:35:52.805882 26090 raft_consensus.cc:348] T 34d072cff51f4900ad7185b674e86d05 P 3fec1956565c4b87b1fe9937797146b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac98ca197074cf881f307af40955cf1" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 46537 } } peers { permanent_uuid: "352297845b9640cd915bb38ffc2c2d32" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 36239 } } peers { permanent_uuid: "3fec1956565c4b87b1fe9937797146b6" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 46669 } }
I20260926 16:35:52.806357 26090 raft_consensus.cc:374] T 34d072cff51f4900ad7185b674e86d05 P 3fec1956565c4b87b1fe9937797146b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:35:52.806442 26090 raft_consensus.cc:729] T 34d072cff51f4900ad7185b674e86d05 P 3fec1956565c4b87b1fe9937797146b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fec1956565c4b87b1fe9937797146b6, State: Initialized, Role: FOLLOWER
I20260926 16:35:52.807024 26090 consensus_queue.cc:260] T 34d072cff51f4900ad7185b674e86d05 P 3fec1956565c4b87b1fe9937797146b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac98ca197074cf881f307af40955cf1" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 46537 } } peers { permanent_uuid: "352297845b9640cd915bb38ffc2c2d32" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 36239 } } peers { permanent_uuid: "3fec1956565c4b87b1fe9937797146b6" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 46669 } }
I20260926 16:35:52.809684 25936 heartbeater.cc:503] Master 127.6.33.62:36817 was elected leader, sending a full tablet report...
I20260926 16:35:52.810180 26090 ts_tablet_manager.cc:1428] T 34d072cff51f4900ad7185b674e86d05 P 3fec1956565c4b87b1fe9937797146b6: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260926 16:35:52.809939 26091 raft_consensus.cc:348] T 34d072cff51f4900ad7185b674e86d05 P 352297845b9640cd915bb38ffc2c2d32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac98ca197074cf881f307af40955cf1" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 46537 } } peers { permanent_uuid: "352297845b9640cd915bb38ffc2c2d32" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 36239 } } peers { permanent_uuid: "3fec1956565c4b87b1fe9937797146b6" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 46669 } }
I20260926 16:35:52.810398 26091 raft_consensus.cc:374] T 34d072cff51f4900ad7185b674e86d05 P 352297845b9640cd915bb38ffc2c2d32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:35:52.810496 26091 raft_consensus.cc:729] T 34d072cff51f4900ad7185b674e86d05 P 352297845b9640cd915bb38ffc2c2d32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 352297845b9640cd915bb38ffc2c2d32, State: Initialized, Role: FOLLOWER
I20260926 16:35:52.811192 26091 consensus_queue.cc:260] T 34d072cff51f4900ad7185b674e86d05 P 352297845b9640cd915bb38ffc2c2d32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac98ca197074cf881f307af40955cf1" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 46537 } } peers { permanent_uuid: "352297845b9640cd915bb38ffc2c2d32" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 36239 } } peers { permanent_uuid: "3fec1956565c4b87b1fe9937797146b6" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 46669 } }
I20260926 16:35:52.813174 26091 ts_tablet_manager.cc:1428] T 34d072cff51f4900ad7185b674e86d05 P 352297845b9640cd915bb38ffc2c2d32: Time spent starting tablet: real 0.014s user 0.012s sys 0.002s
W20260926 16:35:52.865187 25803 tablet.cc:2369] T 34d072cff51f4900ad7185b674e86d05 P 352297845b9640cd915bb38ffc2c2d32: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 16:35:52.895628 25937 tablet.cc:2369] T 34d072cff51f4900ad7185b674e86d05 P 3fec1956565c4b87b1fe9937797146b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 16:35:52.918648 26071 tablet.cc:2369] T 34d072cff51f4900ad7185b674e86d05 P bac98ca197074cf881f307af40955cf1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:35:53.209163 26096 raft_consensus.cc:482] T 34d072cff51f4900ad7185b674e86d05 P 3fec1956565c4b87b1fe9937797146b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:35:53.209452 26096 raft_consensus.cc:504] T 34d072cff51f4900ad7185b674e86d05 P 3fec1956565c4b87b1fe9937797146b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac98ca197074cf881f307af40955cf1" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 46537 } } peers { permanent_uuid: "352297845b9640cd915bb38ffc2c2d32" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 36239 } } peers { permanent_uuid: "3fec1956565c4b87b1fe9937797146b6" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 46669 } }
I20260926 16:35:53.211076 26096 leader_election.cc:288] T 34d072cff51f4900ad7185b674e86d05 P 3fec1956565c4b87b1fe9937797146b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bac98ca197074cf881f307af40955cf1 (127.6.33.3:46537), 352297845b9640cd915bb38ffc2c2d32 (127.6.33.1:36239)
I20260926 16:35:53.218248 26084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:53.211504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:51013 (local address 127.6.33.3:46537)
0926 16:35:53.211736 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:53.211743 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:53.212346 (+ 603us) server_negotiation.cc:407] Connection header received
0926 16:35:53.212706 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:53.212718 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:53.212854 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:53.212971 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:35:53.213597 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:53.214715 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:53.216440 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:53.217315 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:53.217465 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:53.217574 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:53.217854 (+ 280us) server_negotiation.cc:299] Negotiation successful
0926 16:35:53.217981 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 16:35:53.218318 26098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:53.211448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.3:46537 (local address 127.6.33.2:51013)
0926 16:35:53.212078 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:53.212095 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 16:35:53.212463 (+ 368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:35:53.213018 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:35:53.213023 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:35:53.213065 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:35:53.213445 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:53.213459 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:53.214851 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:53.214855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:53.216278 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:53.216290 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:53.217520 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:53.217526 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:53.217695 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:53.217719 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 16:35:53.217817 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 16:35:53.217959 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":311,"threads_started":1}
I20260926 16:35:53.218667 26088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:53.212632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:37823 (local address 127.6.33.1:36239)
0926 16:35:53.212860 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:53.212865 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:53.213370 (+ 505us) server_negotiation.cc:407] Connection header received
0926 16:35:53.213493 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:53.213501 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:53.213590 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:53.213665 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:35:53.214323 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:53.215232 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:53.216961 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:53.217730 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:53.217886 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:53.217994 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:53.218276 (+ 282us) server_negotiation.cc:299] Negotiation successful
0926 16:35:53.218409 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 16:35:53.218726 26099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:53.212287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:36239 (local address 127.6.33.2:37823)
0926 16:35:53.213260 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:53.213275 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 16:35:53.213378 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:35:53.213724 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:35:53.213729 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:35:53.213753 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:35:53.214119 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:53.214130 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:53.215362 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:53.215368 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:53.216740 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:53.216799 (+ 59us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:53.217955 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:53.217961 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:53.218124 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:53.218153 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 16:35:53.218249 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 16:35:53.218380 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":848,"thread_start_us":574,"threads_started":1}
I20260926 16:35:53.219585 26025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34d072cff51f4900ad7185b674e86d05" candidate_uuid: "3fec1956565c4b87b1fe9937797146b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bac98ca197074cf881f307af40955cf1" is_pre_election: true
I20260926 16:35:53.220078 26025 raft_consensus.cc:2454] T 34d072cff51f4900ad7185b674e86d05 P bac98ca197074cf881f307af40955cf1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fec1956565c4b87b1fe9937797146b6 in term 0.
I20260926 16:35:53.219988 25757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34d072cff51f4900ad7185b674e86d05" candidate_uuid: "3fec1956565c4b87b1fe9937797146b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "352297845b9640cd915bb38ffc2c2d32" is_pre_election: true
I20260926 16:35:53.220623 25757 raft_consensus.cc:2454] T 34d072cff51f4900ad7185b674e86d05 P 352297845b9640cd915bb38ffc2c2d32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fec1956565c4b87b1fe9937797146b6 in term 0.
I20260926 16:35:53.220724 25825 leader_election.cc:302] T 34d072cff51f4900ad7185b674e86d05 P 3fec1956565c4b87b1fe9937797146b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fec1956565c4b87b1fe9937797146b6, bac98ca197074cf881f307af40955cf1; no voters:
I20260926 16:35:53.221210 26096 raft_consensus.cc:2781] T 34d072cff51f4900ad7185b674e86d05 P 3fec1956565c4b87b1fe9937797146b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:35:53.221381 26096 raft_consensus.cc:482] T 34d072cff51f4900ad7185b674e86d05 P 3fec1956565c4b87b1fe9937797146b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:35:53.221501 26096 raft_consensus.cc:3037] T 34d072cff51f4900ad7185b674e86d05 P 3fec1956565c4b87b1fe9937797146b6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:35:53.223703 26096 raft_consensus.cc:504] T 34d072cff51f4900ad7185b674e86d05 P 3fec1956565c4b87b1fe9937797146b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac98ca197074cf881f307af40955cf1" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 46537 } } peers { permanent_uuid: "352297845b9640cd915bb38ffc2c2d32" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 36239 } } peers { permanent_uuid: "3fec1956565c4b87b1fe9937797146b6" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 46669 } }
I20260926 16:35:53.224287 26096 leader_election.cc:288] T 34d072cff51f4900ad7185b674e86d05 P 3fec1956565c4b87b1fe9937797146b6 [CANDIDATE]: Term 1 election: Requested vote from peers bac98ca197074cf881f307af40955cf1 (127.6.33.3:46537), 352297845b9640cd915bb38ffc2c2d32 (127.6.33.1:36239)
I20260926 16:35:53.224876 26025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34d072cff51f4900ad7185b674e86d05" candidate_uuid: "3fec1956565c4b87b1fe9937797146b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bac98ca197074cf881f307af40955cf1"
I20260926 16:35:53.224911 25757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34d072cff51f4900ad7185b674e86d05" candidate_uuid: "3fec1956565c4b87b1fe9937797146b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "352297845b9640cd915bb38ffc2c2d32"
I20260926 16:35:53.225081 26025 raft_consensus.cc:3037] T 34d072cff51f4900ad7185b674e86d05 P bac98ca197074cf881f307af40955cf1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:35:53.225117 25757 raft_consensus.cc:3037] T 34d072cff51f4900ad7185b674e86d05 P 352297845b9640cd915bb38ffc2c2d32 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:35:53.227233 26025 raft_consensus.cc:2454] T 34d072cff51f4900ad7185b674e86d05 P bac98ca197074cf881f307af40955cf1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fec1956565c4b87b1fe9937797146b6 in term 1.
I20260926 16:35:53.227454 25757 raft_consensus.cc:2454] T 34d072cff51f4900ad7185b674e86d05 P 352297845b9640cd915bb38ffc2c2d32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fec1956565c4b87b1fe9937797146b6 in term 1.
I20260926 16:35:53.227697 25825 leader_election.cc:302] T 34d072cff51f4900ad7185b674e86d05 P 3fec1956565c4b87b1fe9937797146b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fec1956565c4b87b1fe9937797146b6, bac98ca197074cf881f307af40955cf1; no voters:
I20260926 16:35:53.228001 26096 raft_consensus.cc:2781] T 34d072cff51f4900ad7185b674e86d05 P 3fec1956565c4b87b1fe9937797146b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:35:53.228421 26096 raft_consensus.cc:686] T 34d072cff51f4900ad7185b674e86d05 P 3fec1956565c4b87b1fe9937797146b6 [term 1 LEADER]: Becoming Leader. State: Replica: 3fec1956565c4b87b1fe9937797146b6, State: Running, Role: LEADER
I20260926 16:35:53.228955 26096 consensus_queue.cc:237] T 34d072cff51f4900ad7185b674e86d05 P 3fec1956565c4b87b1fe9937797146b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac98ca197074cf881f307af40955cf1" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 46537 } } peers { permanent_uuid: "352297845b9640cd915bb38ffc2c2d32" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 36239 } } peers { permanent_uuid: "3fec1956565c4b87b1fe9937797146b6" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 46669 } }
I20260926 16:35:53.233397 25612 catalog_manager.cc:5495] T 34d072cff51f4900ad7185b674e86d05 P 3fec1956565c4b87b1fe9937797146b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fec1956565c4b87b1fe9937797146b6 (127.6.33.2). New cstate: current_term: 1 leader_uuid: "3fec1956565c4b87b1fe9937797146b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac98ca197074cf881f307af40955cf1" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 46537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "352297845b9640cd915bb38ffc2c2d32" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 36239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fec1956565c4b87b1fe9937797146b6" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 46669 } health_report { overall_health: HEALTHY } } }
I20260926 16:35:53.325464 26104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:53.319557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54536 (local address 127.6.33.2:46669)
0926 16:35:53.320254 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:53.320260 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:53.320282 (+ 22us) server_negotiation.cc:407] Connection header received
0926 16:35:53.320379 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:53.320383 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:53.320460 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:53.320640 (+ 180us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:35:53.321174 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:53.321905 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:53.323317 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:53.323708 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:53.323834 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:53.324292 (+ 458us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:35:53.324878 (+ 586us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:35:53.325049 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:53.325101 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 16:35:53.325183 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":446,"threads_started":1}
I20260926 16:35:53.333489 25757 raft_consensus.cc:1261] T 34d072cff51f4900ad7185b674e86d05 P 352297845b9640cd915bb38ffc2c2d32 [term 1 FOLLOWER]: Refusing update from remote peer 3fec1956565c4b87b1fe9937797146b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:35:53.333491 26025 raft_consensus.cc:1261] T 34d072cff51f4900ad7185b674e86d05 P bac98ca197074cf881f307af40955cf1 [term 1 FOLLOWER]: Refusing update from remote peer 3fec1956565c4b87b1fe9937797146b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:35:53.334458 26100 consensus_queue.cc:1035] T 34d072cff51f4900ad7185b674e86d05 P 3fec1956565c4b87b1fe9937797146b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "352297845b9640cd915bb38ffc2c2d32" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 36239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:35:53.334847 26096 consensus_queue.cc:1035] T 34d072cff51f4900ad7185b674e86d05 P 3fec1956565c4b87b1fe9937797146b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bac98ca197074cf881f307af40955cf1" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 46537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:35:53.352236 26108 mvcc.cc:205] Tried to move back new op lower bound from 7333644506443837440 to 7333644506029539328. Current Snapshot: MvccSnapshot[applied={T|T < 7333644506443837440}]
I20260926 16:35:53.363515 25613 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43702:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:35:53.363842 25613 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43702:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:35:53.367730 25613 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 896c4f0c5de6461fa12759890e9886a3: Sending DeleteTablet for 3 replicas of tablet 34d072cff51f4900ad7185b674e86d05
I20260926 16:35:53.369282 25871 tablet_service.cc:1514] Processing DeleteTablet for tablet 34d072cff51f4900ad7185b674e86d05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:35:53 UTC) from {username='kudu', principal='kudu/127.6.33.62@KRBTEST.COM'} at 127.0.0.1:54520
I20260926 16:35:53.369478 26005 tablet_service.cc:1514] Processing DeleteTablet for tablet 34d072cff51f4900ad7185b674e86d05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:35:53 UTC) from {username='kudu', principal='kudu/127.6.33.62@KRBTEST.COM'} at 127.0.0.1:46214
I20260926 16:35:53.369550 25737 tablet_service.cc:1514] Processing DeleteTablet for tablet 34d072cff51f4900ad7185b674e86d05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:35:53 UTC) from {username='kudu', principal='kudu/127.6.33.62@KRBTEST.COM'} at 127.0.0.1:50880
I20260926 16:35:53.370872 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 25674
I20260926 16:35:53.380398 26119 tablet_replica.cc:330] T 34d072cff51f4900ad7185b674e86d05 P 352297845b9640cd915bb38ffc2c2d32: stopping tablet replica
I20260926 16:35:53.381078 26119 raft_consensus.cc:2229] T 34d072cff51f4900ad7185b674e86d05 P 352297845b9640cd915bb38ffc2c2d32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:35:53.381778 26119 raft_consensus.cc:2258] T 34d072cff51f4900ad7185b674e86d05 P 352297845b9640cd915bb38ffc2c2d32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:35:53.382120 26117 tablet_replica.cc:330] T 34d072cff51f4900ad7185b674e86d05 P 3fec1956565c4b87b1fe9937797146b6: stopping tablet replica
I20260926 16:35:53.382675 26117 raft_consensus.cc:2229] T 34d072cff51f4900ad7185b674e86d05 P 3fec1956565c4b87b1fe9937797146b6 [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:35:53.383523 26117 raft_consensus.cc:2258] T 34d072cff51f4900ad7185b674e86d05 P 3fec1956565c4b87b1fe9937797146b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:35:53.385834 26119 ts_tablet_manager.cc:1905] T 34d072cff51f4900ad7185b674e86d05 P 352297845b9640cd915bb38ffc2c2d32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:35:53.385836 26117 ts_tablet_manager.cc:1905] T 34d072cff51f4900ad7185b674e86d05 P 3fec1956565c4b87b1fe9937797146b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:35:53.389619 26118 tablet_replica.cc:330] T 34d072cff51f4900ad7185b674e86d05 P bac98ca197074cf881f307af40955cf1: stopping tablet replica
I20260926 16:35:53.390064 26118 raft_consensus.cc:2229] T 34d072cff51f4900ad7185b674e86d05 P bac98ca197074cf881f307af40955cf1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:35:53.390650 26118 raft_consensus.cc:2258] T 34d072cff51f4900ad7185b674e86d05 P bac98ca197074cf881f307af40955cf1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:35:53.394547 26117 ts_tablet_manager.cc:1918] T 34d072cff51f4900ad7185b674e86d05 P 3fec1956565c4b87b1fe9937797146b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:35:53.394781 26117 log.cc:1196] T 34d072cff51f4900ad7185b674e86d05 P 3fec1956565c4b87b1fe9937797146b6: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/ts-1/wal/wals/34d072cff51f4900ad7185b674e86d05
I20260926 16:35:53.395378 26117 ts_tablet_manager.cc:1939] T 34d072cff51f4900ad7185b674e86d05 P 3fec1956565c4b87b1fe9937797146b6: Deleting consensus metadata
I20260926 16:35:53.397488 26119 ts_tablet_manager.cc:1918] T 34d072cff51f4900ad7185b674e86d05 P 352297845b9640cd915bb38ffc2c2d32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:35:53.397451 25599 catalog_manager.cc:4841] TS 3fec1956565c4b87b1fe9937797146b6 (127.6.33.2:46669): tablet 34d072cff51f4900ad7185b674e86d05 (table test-table [id=88832550bb464254add3c79f5a697a17]) successfully deleted
Sep 26 16:35:53 dist-test-slave-khd3 krb5kdc[25568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.1@KRBTEST.COM
I20260926 16:35:53.399164 26119 log.cc:1196] T 34d072cff51f4900ad7185b674e86d05 P 352297845b9640cd915bb38ffc2c2d32: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/ts-0/wal/wals/34d072cff51f4900ad7185b674e86d05
I20260926 16:35:53.399271 26118 ts_tablet_manager.cc:1905] T 34d072cff51f4900ad7185b674e86d05 P bac98ca197074cf881f307af40955cf1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:35:53.399772 26119 ts_tablet_manager.cc:1939] T 34d072cff51f4900ad7185b674e86d05 P 352297845b9640cd915bb38ffc2c2d32: Deleting consensus metadata
I20260926 16:35:53.401767 25598 catalog_manager.cc:4841] TS 352297845b9640cd915bb38ffc2c2d32 (127.6.33.1:36239): tablet 34d072cff51f4900ad7185b674e86d05 (table test-table [id=88832550bb464254add3c79f5a697a17]) successfully deleted
I20260926 16:35:53.406172 26118 ts_tablet_manager.cc:1918] T 34d072cff51f4900ad7185b674e86d05 P bac98ca197074cf881f307af40955cf1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:35:53.406441 26118 log.cc:1196] T 34d072cff51f4900ad7185b674e86d05 P bac98ca197074cf881f307af40955cf1: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440233850748-6276-0/minicluster-data/ts-2/wal/wals/34d072cff51f4900ad7185b674e86d05
I20260926 16:35:53.406996 26088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:53.372269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50894 (local address 127.6.33.1:36239)
0926 16:35:53.372455 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:53.372462 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:53.386347 (+ 13885us) server_negotiation.cc:407] Connection header received
0926 16:35:53.386659 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:53.386665 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:53.386768 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:53.386853 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:35:53.388894 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:53.389994 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:53.392159 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:53.392956 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:53.393072 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:53.399117 (+ 6045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:35:53.399135 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:35:53.399158 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:35:53.399210 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:35:53.403257 (+ 4047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:35:53.404198 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:35:53.404213 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:35:53.404217 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:35:53.404394 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:35:53.404913 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:35:53.404919 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:35:53.404923 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:35:53.405434 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:35:53.405808 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:53.406212 (+ 404us) server_negotiation.cc:299] Negotiation successful
0926 16:35:53.406472 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 16:35:53.407068 26118 ts_tablet_manager.cc:1939] T 34d072cff51f4900ad7185b674e86d05 P bac98ca197074cf881f307af40955cf1: Deleting consensus metadata
I20260926 16:35:53.407758 25797 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:35:53.409183 25598 catalog_manager.cc:4841] TS bac98ca197074cf881f307af40955cf1 (127.6.33.3:46537): tablet 34d072cff51f4900ad7185b674e86d05 (table test-table [id=88832550bb464254add3c79f5a697a17]) successfully deleted
I20260926 16:35:53.499173 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 25674
I20260926 16:35:53.517477 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 25808
Sep 26 16:35:53 dist-test-slave-khd3 krb5kdc[25568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.2@KRBTEST.COM
I20260926 16:35:53.537920 26104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:53.518295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54542 (local address 127.6.33.2:46669)
0926 16:35:53.518472 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:53.518480 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:53.518507 (+ 27us) server_negotiation.cc:407] Connection header received
0926 16:35:53.518730 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:53.518735 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:53.518813 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:53.518920 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:35:53.520166 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:53.521031 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:53.522442 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:53.522883 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:53.523025 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:53.531445 (+ 8420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:35:53.531462 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:35:53.531478 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:35:53.531527 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:35:53.534730 (+ 3203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:35:53.535497 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:35:53.535506 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:35:53.535508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:35:53.535683 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:35:53.536085 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:35:53.536091 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:35:53.536095 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:35:53.536559 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:35:53.536916 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:53.537202 (+ 286us) server_negotiation.cc:299] Negotiation successful
0926 16:35:53.537468 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 16:35:53.538793 25931 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:35:53.678355 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 25808
I20260926 16:35:53.699783 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 25942
Sep 26 16:35:53 dist-test-slave-khd3 krb5kdc[25568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.3@KRBTEST.COM
I20260926 16:35:53.715498 26084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:53.700565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46218 (local address 127.6.33.3:46537)
0926 16:35:53.700780 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:53.700789 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:53.700813 (+ 24us) server_negotiation.cc:407] Connection header received
0926 16:35:53.701115 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:53.701119 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:53.701202 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:53.701326 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:35:53.702656 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:53.703410 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:53.704830 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:53.705234 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:53.705367 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:53.709793 (+ 4426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:35:53.709809 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:35:53.709825 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:35:53.709862 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:35:53.712729 (+ 2867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:35:53.713540 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:35:53.713548 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:35:53.713551 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:35:53.713693 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:35:53.714033 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:35:53.714037 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:35:53.714039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:35:53.714410 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:35:53.714638 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:53.714929 (+ 291us) server_negotiation.cc:299] Negotiation successful
0926 16:35:53.715175 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 16:35:53.716224 26065 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:35:53.805963 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 25942
I20260926 16:35:53.823055 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 25581
I20260926 16:35:53.824139 25643 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:35:53.935846 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 25581
2026-09-26T16:35:53Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4834 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-26T16:35:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:35:53Z Disabled control of system clock
I20260926 16:35:54.015726 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:39461
--webserver_interface=127.6.33.62
--webserver_port=0
--builtin_ntp_servers=127.6.33.20:37719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.33.62:39461
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 16:35:54.363559 26128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:35:54.363905 26128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:35:54.364053 26128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:35:54.371434 26128 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:35:54.371603 26128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:35:54.371690 26128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:35:54.371793 26128 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:35:54.371872 26128 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:35:54.381451 26128 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:37719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.33.62:39461
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:39461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:35:54.382899 26128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:35:54.384801 26128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:35:54.396986 26133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:35:54.396955 26136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:35:54.397665 26134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:35:54.398021 26128 server_base.cc:1034] running on GCE node
I20260926 16:35:54.398845 26128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:35:54.400295 26128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:35:54.401510 26128 hybrid_clock.cc:648] HybridClock initialized: now 1790440554401431 us; error 77 us; skew 500 ppm
I20260926 16:35:54.401858 26128 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:35:54.404155 26128 webserver.cc:460] Webserver started at http://127.6.33.62:33501/ using document root <none> and password file <none>
I20260926 16:35:54.404910 26128 fs_manager.cc:359] Metadata directory not provided
I20260926 16:35:54.405033 26128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:35:54.405357 26128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:35:54.408233 26128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/master-0/data/instance:
uuid: "1ddb40f5d77347d091334b51691ec041"
format_stamp: "Formatted at 2026-09-26 16:35:54 on dist-test-slave-khd3"
server_key: "a8be644713f7f27710f678e6687ca864"
server_key_iv: "dc102e48aa99076cc1fc4aba3b7fb4b5"
server_key_version: "encryptionkey@0"
I20260926 16:35:54.409104 26128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/master-0/wal/instance:
uuid: "1ddb40f5d77347d091334b51691ec041"
format_stamp: "Formatted at 2026-09-26 16:35:54 on dist-test-slave-khd3"
server_key: "a8be644713f7f27710f678e6687ca864"
server_key_iv: "dc102e48aa99076cc1fc4aba3b7fb4b5"
server_key_version: "encryptionkey@0"
I20260926 16:35:54.414196 26128 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260926 16:35:54.417650 26142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:35:54.418977 26128 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:35:54.419183 26128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/master-0/wal
uuid: "1ddb40f5d77347d091334b51691ec041"
format_stamp: "Formatted at 2026-09-26 16:35:54 on dist-test-slave-khd3"
server_key: "a8be644713f7f27710f678e6687ca864"
server_key_iv: "dc102e48aa99076cc1fc4aba3b7fb4b5"
server_key_version: "encryptionkey@0"
I20260926 16:35:54.419400 26128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:35:54.435294 26128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:35:54.436270 26128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:35:54.436583 26128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:35:54.455681 26128 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.62:39461
I20260926 16:35:54.455716 26193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.62:39461 every 8 connection(s)
I20260926 16:35:54.457255 26128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
I20260926 16:35:54.461230 26194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:35:54.464138 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 26128
I20260926 16:35:54.464315 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/master-0/wal/instance
I20260926 16:35:54.464869 6276 external_mini_cluster.cc:1428] Setting key 82944e6d39ddd85d3adc52cc4256824e
I20260926 16:35:54.473379 26194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ddb40f5d77347d091334b51691ec041: Bootstrap starting.
I20260926 16:35:54.476609 26194 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ddb40f5d77347d091334b51691ec041: Neither blocks nor log segments found. Creating new log.
I20260926 16:35:54.477802 26194 log.cc:824] T 00000000000000000000000000000000 P 1ddb40f5d77347d091334b51691ec041: Log is configured to *not* fsync() on all Append() calls
I20260926 16:35:54.478664 26197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:54.466935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48682 (local address 127.6.33.62:39461)
0926 16:35:54.467947 (+ 1012us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:54.467958 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:54.467997 (+ 39us) server_negotiation.cc:407] Connection header received
0926 16:35:54.469761 (+ 1764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:54.469785 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:54.470253 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:54.470658 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:35:54.471243 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:54.472547 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:54.474021 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:54.474724 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:54.474927 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:54.476119 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:35:54.476127 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:35:54.476141 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:35:54.476666 (+ 525us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:35:54.476823 (+ 157us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:35:54.476851 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:35:54.476991 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:54.477195 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 16:35:54.477496 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":358,"threads_started":1}
I20260926 16:35:54.480931 26194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ddb40f5d77347d091334b51691ec041: No bootstrap required, opened a new log
I20260926 16:35:54.488543 26194 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ddb40f5d77347d091334b51691ec041 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ddb40f5d77347d091334b51691ec041" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 39461 } }
I20260926 16:35:54.489168 26194 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ddb40f5d77347d091334b51691ec041 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:35:54.489295 26194 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ddb40f5d77347d091334b51691ec041 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ddb40f5d77347d091334b51691ec041, State: Initialized, Role: FOLLOWER
I20260926 16:35:54.490087 26194 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ddb40f5d77347d091334b51691ec041 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ddb40f5d77347d091334b51691ec041" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 39461 } }
I20260926 16:35:54.490368 26194 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ddb40f5d77347d091334b51691ec041 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:35:54.490532 26194 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ddb40f5d77347d091334b51691ec041 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:35:54.490748 26194 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ddb40f5d77347d091334b51691ec041 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:35:54.493094 26194 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ddb40f5d77347d091334b51691ec041 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ddb40f5d77347d091334b51691ec041" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 39461 } }
I20260926 16:35:54.493613 26194 leader_election.cc:302] T 00000000000000000000000000000000 P 1ddb40f5d77347d091334b51691ec041 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ddb40f5d77347d091334b51691ec041; no voters:
I20260926 16:35:54.494236 26194 leader_election.cc:288] T 00000000000000000000000000000000 P 1ddb40f5d77347d091334b51691ec041 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:35:54.494393 26199 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ddb40f5d77347d091334b51691ec041 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:35:54.495069 26199 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ddb40f5d77347d091334b51691ec041 [term 1 LEADER]: Becoming Leader. State: Replica: 1ddb40f5d77347d091334b51691ec041, State: Running, Role: LEADER
I20260926 16:35:54.495738 26199 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ddb40f5d77347d091334b51691ec041 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ddb40f5d77347d091334b51691ec041" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 39461 } }
I20260926 16:35:54.496327 26194 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ddb40f5d77347d091334b51691ec041 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:35:54.498701 26200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ddb40f5d77347d091334b51691ec041 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ddb40f5d77347d091334b51691ec041" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ddb40f5d77347d091334b51691ec041" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 39461 } } }
I20260926 16:35:54.499172 26200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ddb40f5d77347d091334b51691ec041 [sys.catalog]: This master's current role is: LEADER
I20260926 16:35:54.498679 26201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ddb40f5d77347d091334b51691ec041 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ddb40f5d77347d091334b51691ec041. Latest consensus state: current_term: 1 leader_uuid: "1ddb40f5d77347d091334b51691ec041" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ddb40f5d77347d091334b51691ec041" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 39461 } } }
I20260926 16:35:54.499529 26201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ddb40f5d77347d091334b51691ec041 [sys.catalog]: This master's current role is: LEADER
I20260926 16:35:54.500054 26207 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:35:54.506477 26207 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:35:54.519477 26207 catalog_manager.cc:1347] Generated new cluster ID: c4b13efae2cf4c67844b43593cca2d5d
I20260926 16:35:54.519619 26207 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:35:54.531514 26207 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:35:54.533493 26207 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:35:54.541556 26207 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ddb40f5d77347d091334b51691ec041: Generated new TSK 0
I20260926 16:35:54.542656 26207 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:35:54.552742 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.1:0
--local_ip_for_outbound_sockets=127.6.33.1
--webserver_interface=127.6.33.1
--webserver_port=0
--tserver_master_addrs=127.6.33.62:39461
--builtin_ntp_servers=127.6.33.20:37719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 16:35:54.932655 26218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:35:54.932991 26218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:35:54.933081 26218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:35:54.940307 26218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:35:54.940418 26218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:35:54.940533 26218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.1
I20260926 16:35:54.949842 26218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:37719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.33.1
--webserver_port=0
--tserver_master_addrs=127.6.33.62:39461
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.1
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:35:54.951130 26218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:35:54.952946 26218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:35:54.966432 26223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:35:54.966400 26226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:35:54.966358 26224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:35:54.967331 26218 server_base.cc:1034] running on GCE node
I20260926 16:35:54.968057 26218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:35:54.969064 26218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:35:54.970285 26218 hybrid_clock.cc:648] HybridClock initialized: now 1790440554970215 us; error 58 us; skew 500 ppm
I20260926 16:35:54.970620 26218 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:35:54.972951 26218 webserver.cc:460] Webserver started at http://127.6.33.1:37115/ using document root <none> and password file <none>
I20260926 16:35:54.973829 26218 fs_manager.cc:359] Metadata directory not provided
I20260926 16:35:54.973940 26218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:35:54.974278 26218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:35:54.977375 26218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/ts-0/data/instance:
uuid: "47ee40ef8df74033b84845a19217ad43"
format_stamp: "Formatted at 2026-09-26 16:35:54 on dist-test-slave-khd3"
server_key: "85e6e8f6bc339a1dcf1e96c04669ed03"
server_key_iv: "f25a4f32dead53d19d7c772a4e1f1f71"
server_key_version: "encryptionkey@0"
I20260926 16:35:54.978441 26218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/ts-0/wal/instance:
uuid: "47ee40ef8df74033b84845a19217ad43"
format_stamp: "Formatted at 2026-09-26 16:35:54 on dist-test-slave-khd3"
server_key: "85e6e8f6bc339a1dcf1e96c04669ed03"
server_key_iv: "f25a4f32dead53d19d7c772a4e1f1f71"
server_key_version: "encryptionkey@0"
I20260926 16:35:54.984272 26218 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 16:35:54.987906 26232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:35:54.989674 26218 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:35:54.989861 26218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/ts-0/wal
uuid: "47ee40ef8df74033b84845a19217ad43"
format_stamp: "Formatted at 2026-09-26 16:35:54 on dist-test-slave-khd3"
server_key: "85e6e8f6bc339a1dcf1e96c04669ed03"
server_key_iv: "f25a4f32dead53d19d7c772a4e1f1f71"
server_key_version: "encryptionkey@0"
I20260926 16:35:54.990186 26218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:35:55.006939 26218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:35:55.007941 26218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:35:55.008225 26218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:35:55.009372 26218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:35:55.010948 26218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:35:55.011040 26218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:35:55.011118 26218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:35:55.011173 26218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:35:55.046773 26218 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.1:46097
I20260926 16:35:55.046826 26344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.1:46097 every 8 connection(s)
I20260926 16:35:55.048086 26218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/ts-0/data/info.pb
I20260926 16:35:55.049618 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 26218
I20260926 16:35:55.049799 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/ts-0/wal/instance
I20260926 16:35:55.050292 6276 external_mini_cluster.cc:1428] Setting key afccc2dc9619b037e534bcea6c43c729
I20260926 16:35:55.053259 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.2:0
--local_ip_for_outbound_sockets=127.6.33.2
--webserver_interface=127.6.33.2
--webserver_port=0
--tserver_master_addrs=127.6.33.62:39461
--builtin_ntp_servers=127.6.33.20:37719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 16:35:55.062350 26347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:55.050893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:33287 (local address 127.6.33.62:39461)
0926 16:35:55.051403 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:55.051409 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:55.052998 (+ 1589us) server_negotiation.cc:407] Connection header received
0926 16:35:55.054935 (+ 1937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:55.054944 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:55.055041 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:55.055181 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:35:55.056345 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:55.057542 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:55.059161 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:55.059902 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:55.060054 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:55.061548 (+ 1494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:35:55.061553 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:35:55.061556 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:35:55.061575 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:35:55.061638 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:35:55.061651 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:35:55.061721 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:55.061999 (+ 278us) server_negotiation.cc:299] Negotiation successful
0926 16:35:55.062085 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":250,"threads_started":1}
I20260926 16:35:55.063319 26348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:55.051658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:39461 (local address 127.6.33.1:33287)
0926 16:35:55.052821 (+ 1163us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:55.052877 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 16:35:55.054641 (+ 1764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:35:55.055426 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:35:55.055434 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:35:55.055552 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:35:55.056166 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:55.056183 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:55.057727 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:55.057731 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:55.059001 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:55.059015 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:55.060127 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:55.060133 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:55.060337 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:55.061175 (+ 838us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:35:55.061180 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:35:55.061238 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:35:55.061351 (+ 113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:35:55.061744 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:35:55.061758 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:35:55.061768 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 16:35:55.061939 (+ 171us) client_negotiation.cc:240] Negotiation successful
0926 16:35:55.062254 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":377,"threads_started":1}
I20260926 16:35:55.064881 26345 heartbeater.cc:347] Connected to a master server at 127.6.33.62:39461
I20260926 16:35:55.065207 26345 heartbeater.cc:464] Registering TS with master...
I20260926 16:35:55.066164 26345 heartbeater.cc:511] Master 127.6.33.62:39461 requested a full tablet report, sending...
I20260926 16:35:55.068326 26159 ts_manager.cc:194] Registered new tserver with Master: 47ee40ef8df74033b84845a19217ad43 (127.6.33.1:46097)
I20260926 16:35:55.069829 26159 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.33.1:33287
W20260926 16:35:55.385499 26349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:35:55.385820 26349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:35:55.385973 26349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:35:55.393589 26349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:35:55.393700 26349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:35:55.393875 26349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.2
I20260926 16:35:55.403658 26349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:37719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.33.2
--webserver_port=0
--tserver_master_addrs=127.6.33.62:39461
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.2
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:35:55.405030 26349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:35:55.406857 26349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:35:55.420181 26355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:35:55.420138 26354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:35:55.420395 26357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:35:55.420820 26349 server_base.cc:1034] running on GCE node
I20260926 16:35:55.421465 26349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:35:55.422516 26349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:35:55.423733 26349 hybrid_clock.cc:648] HybridClock initialized: now 1790440555423664 us; error 63 us; skew 500 ppm
I20260926 16:35:55.424083 26349 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:35:55.426704 26349 webserver.cc:460] Webserver started at http://127.6.33.2:40343/ using document root <none> and password file <none>
I20260926 16:35:55.427476 26349 fs_manager.cc:359] Metadata directory not provided
I20260926 16:35:55.427614 26349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:35:55.427960 26349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:35:55.431161 26349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/ts-1/data/instance:
uuid: "e6b4bfa6b3c94d8db962bbe5d44cf8b7"
format_stamp: "Formatted at 2026-09-26 16:35:55 on dist-test-slave-khd3"
server_key: "7f7c4474824ebe3a4d05c5ea636a4ae5"
server_key_iv: "4e92746025bd090531e663a1c02aa1d1"
server_key_version: "encryptionkey@0"
I20260926 16:35:55.431988 26349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/ts-1/wal/instance:
uuid: "e6b4bfa6b3c94d8db962bbe5d44cf8b7"
format_stamp: "Formatted at 2026-09-26 16:35:55 on dist-test-slave-khd3"
server_key: "7f7c4474824ebe3a4d05c5ea636a4ae5"
server_key_iv: "4e92746025bd090531e663a1c02aa1d1"
server_key_version: "encryptionkey@0"
I20260926 16:35:55.437551 26349 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260926 16:35:55.441550 26363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:35:55.443168 26349 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260926 16:35:55.443377 26349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/ts-1/wal
uuid: "e6b4bfa6b3c94d8db962bbe5d44cf8b7"
format_stamp: "Formatted at 2026-09-26 16:35:55 on dist-test-slave-khd3"
server_key: "7f7c4474824ebe3a4d05c5ea636a4ae5"
server_key_iv: "4e92746025bd090531e663a1c02aa1d1"
server_key_version: "encryptionkey@0"
I20260926 16:35:55.443624 26349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:35:55.484293 26349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:35:55.485261 26349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:35:55.485562 26349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:35:55.486589 26349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:35:55.488204 26349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:35:55.488330 26349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:35:55.488453 26349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:35:55.488538 26349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:35:55.521678 26349 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.2:43853
I20260926 16:35:55.521764 26475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.2:43853 every 8 connection(s)
I20260926 16:35:55.523061 26349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/ts-1/data/info.pb
I20260926 16:35:55.532599 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 26349
I20260926 16:35:55.532830 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/ts-1/wal/instance
I20260926 16:35:55.533308 6276 external_mini_cluster.cc:1428] Setting key 55566e5ea8649410672fefc0494060cf
I20260926 16:35:55.535856 26347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:55.525802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.2:40237 (local address 127.6.33.62:39461)
0926 16:35:55.526007 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:55.526013 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:55.527447 (+ 1434us) server_negotiation.cc:407] Connection header received
0926 16:35:55.529313 (+ 1866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:55.529318 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:55.529388 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:55.529477 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:35:55.530568 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:55.531330 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:55.532966 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:55.533481 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:55.533620 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:55.535069 (+ 1449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:35:55.535075 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:35:55.535080 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:35:55.535104 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:35:55.535135 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:35:55.535148 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:35:55.535238 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:55.535446 (+ 208us) server_negotiation.cc:299] Negotiation successful
0926 16:35:55.535551 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 16:35:55.535970 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.33.3:0
--local_ip_for_outbound_sockets=127.6.33.3
--webserver_interface=127.6.33.3
--webserver_port=0
--tserver_master_addrs=127.6.33.62:39461
--builtin_ntp_servers=127.6.33.20:37719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 16:35:55.536998 26478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:55.526354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:39461 (local address 127.6.33.2:40237)
0926 16:35:55.527287 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:55.527329 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 16:35:55.529049 (+ 1720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:35:55.529734 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:35:55.529742 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:35:55.529837 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:35:55.530391 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:55.530406 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:55.531494 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:55.531499 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:55.532801 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:55.532822 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:55.533711 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:55.533717 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:55.533952 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:55.534752 (+ 800us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:35:55.534755 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:35:55.534794 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:35:55.534876 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:35:55.535238 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:35:55.535251 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:35:55.535261 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 16:35:55.535437 (+ 176us) client_negotiation.cc:240] Negotiation successful
0926 16:35:55.535835 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":298,"threads_started":1}
I20260926 16:35:55.538532 26476 heartbeater.cc:347] Connected to a master server at 127.6.33.62:39461
I20260926 16:35:55.538843 26476 heartbeater.cc:464] Registering TS with master...
I20260926 16:35:55.539606 26476 heartbeater.cc:511] Master 127.6.33.62:39461 requested a full tablet report, sending...
I20260926 16:35:55.541127 26159 ts_manager.cc:194] Registered new tserver with Master: e6b4bfa6b3c94d8db962bbe5d44cf8b7 (127.6.33.2:43853)
I20260926 16:35:55.542467 26159 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.33.2:40237
W20260926 16:35:55.876279 26479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:35:55.876590 26479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:35:55.876677 26479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:35:55.884258 26479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:35:55.884378 26479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:35:55.884490 26479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.33.3
I20260926 16:35:55.893724 26479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:37719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.33.3
--webserver_port=0
--tserver_master_addrs=127.6.33.62:39461
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.33.3
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:35:55.895195 26479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:35:55.897182 26479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:35:55.910660 26484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:35:55.910629 26487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:35:55.910686 26485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:35:55.911841 26479 server_base.cc:1034] running on GCE node
I20260926 16:35:55.912575 26479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:35:55.913655 26479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:35:55.914881 26479 hybrid_clock.cc:648] HybridClock initialized: now 1790440555914818 us; error 58 us; skew 500 ppm
I20260926 16:35:55.915220 26479 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:35:55.917583 26479 webserver.cc:460] Webserver started at http://127.6.33.3:43625/ using document root <none> and password file <none>
I20260926 16:35:55.918280 26479 fs_manager.cc:359] Metadata directory not provided
I20260926 16:35:55.918406 26479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:35:55.918726 26479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:35:55.921597 26479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/ts-2/data/instance:
uuid: "907182b2fba4422ebfa5ad9065af8c4a"
format_stamp: "Formatted at 2026-09-26 16:35:55 on dist-test-slave-khd3"
server_key: "cd77520fc02ddb0bc6c8e47b97eec8f5"
server_key_iv: "484be719a73c90e2e45babd2f56bae69"
server_key_version: "encryptionkey@0"
I20260926 16:35:55.922350 26479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/ts-2/wal/instance:
uuid: "907182b2fba4422ebfa5ad9065af8c4a"
format_stamp: "Formatted at 2026-09-26 16:35:55 on dist-test-slave-khd3"
server_key: "cd77520fc02ddb0bc6c8e47b97eec8f5"
server_key_iv: "484be719a73c90e2e45babd2f56bae69"
server_key_version: "encryptionkey@0"
I20260926 16:35:55.927650 26479 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 16:35:55.931236 26493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:35:55.932655 26479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 16:35:55.932876 26479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/ts-2/wal
uuid: "907182b2fba4422ebfa5ad9065af8c4a"
format_stamp: "Formatted at 2026-09-26 16:35:55 on dist-test-slave-khd3"
server_key: "cd77520fc02ddb0bc6c8e47b97eec8f5"
server_key_iv: "484be719a73c90e2e45babd2f56bae69"
server_key_version: "encryptionkey@0"
I20260926 16:35:55.933130 26479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:35:55.951316 26479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:35:55.952222 26479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:35:55.952464 26479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:35:55.953513 26479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:35:55.954965 26479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:35:55.955061 26479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:35:55.955138 26479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:35:55.955201 26479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:35:55.988543 26479 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.3:34025
I20260926 16:35:55.988639 26605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.3:34025 every 8 connection(s)
I20260926 16:35:55.990024 26479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/ts-2/data/info.pb
I20260926 16:35:55.992463 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 26479
I20260926 16:35:55.992654 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/ts-2/wal/instance
I20260926 16:35:55.993206 6276 external_mini_cluster.cc:1428] Setting key e75d7825ea07f121ece2ce51bdc4e2df
I20260926 16:35:56.003463 26347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:55.992809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.3:57435 (local address 127.6.33.62:39461)
0926 16:35:55.993026 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:55.993032 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:55.994795 (+ 1763us) server_negotiation.cc:407] Connection header received
0926 16:35:55.996874 (+ 2079us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:55.996878 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:55.996957 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:55.997045 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:35:55.998306 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:55.999103 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:56.000606 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:56.001234 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:56.001341 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:56.002733 (+ 1392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:35:56.002739 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:35:56.002742 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:35:56.002763 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:35:56.002799 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:35:56.002813 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:35:56.002902 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:56.003091 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 16:35:56.003176 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 16:35:56.004400 26608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:55.993439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:39461 (local address 127.6.33.3:57435)
0926 16:35:55.994574 (+ 1135us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:55.994677 (+ 103us) client_negotiation.cc:174] Beginning negotiation
0926 16:35:55.996499 (+ 1822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:35:55.997418 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:35:55.997426 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:35:55.997541 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:35:55.998146 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:55.998162 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:55.999229 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:55.999233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:56.000444 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:56.000456 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:56.001433 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:56.001438 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:56.001659 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:56.002407 (+ 748us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:35:56.002410 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:35:56.002449 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:35:56.002531 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:35:56.002906 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:35:56.002913 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:35:56.002919 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 16:35:56.003059 (+ 140us) client_negotiation.cc:240] Negotiation successful
0926 16:35:56.003347 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":375,"threads_started":1}
I20260926 16:35:56.005991 26606 heartbeater.cc:347] Connected to a master server at 127.6.33.62:39461
I20260926 16:35:56.006335 26606 heartbeater.cc:464] Registering TS with master...
I20260926 16:35:56.007277 26606 heartbeater.cc:511] Master 127.6.33.62:39461 requested a full tablet report, sending...
I20260926 16:35:56.008881 26159 ts_manager.cc:194] Registered new tserver with Master: 907182b2fba4422ebfa5ad9065af8c4a (127.6.33.3:34025)
I20260926 16:35:56.009672 26159 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.33.3:57435
I20260926 16:35:56.021067 6276 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:35:56.039846 26347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:56.025321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48690 (local address 127.6.33.62:39461)
0926 16:35:56.025577 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:56.025584 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:56.025908 (+ 324us) server_negotiation.cc:407] Connection header received
0926 16:35:56.026153 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:56.026159 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:56.026246 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:56.026349 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:35:56.026813 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:56.027568 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:56.029017 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:56.029444 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:56.029591 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:56.039042 (+ 9451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:35:56.039048 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:35:56.039052 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:35:56.039076 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:35:56.039115 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:35:56.039130 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:35:56.039235 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:56.039400 (+ 165us) server_negotiation.cc:299] Negotiation successful
0926 16:35:56.039524 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 16:35:56.044202 26159 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48690:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:35:56.053959 26159 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:35:56.073489 26345 heartbeater.cc:503] Master 127.6.33.62:39461 was elected leader, sending a full tablet report...
I20260926 16:35:56.080073 26619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:56.066841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58128 (local address 127.6.33.3:34025)
0926 16:35:56.067740 (+ 899us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:56.067754 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:56.068495 (+ 741us) server_negotiation.cc:407] Connection header received
0926 16:35:56.068948 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:56.068956 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:56.069263 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:56.069430 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:35:56.070555 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:56.072529 (+ 1974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:56.076047 (+ 3518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:56.077966 (+ 1919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:56.078107 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:56.078463 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:56.079319 (+ 856us) server_negotiation.cc:299] Negotiation successful
0926 16:35:56.079550 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":471,"threads_started":1}
I20260926 16:35:56.081089 26618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:56.066669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.3:34025 (local address 127.0.0.1:58128)
0926 16:35:56.068347 (+ 1678us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:56.068389 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 16:35:56.068661 (+ 272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:35:56.069850 (+ 1189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:35:56.069860 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:35:56.069902 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:35:56.070313 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:56.070328 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:56.072744 (+ 2416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:56.072752 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:56.075771 (+ 3019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:56.075791 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:56.078683 (+ 2892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:56.078692 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:56.079002 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:56.079073 (+ 71us) client_negotiation.cc:769] Sending connection context
0926 16:35:56.080336 (+ 1263us) client_negotiation.cc:240] Negotiation successful
0926 16:35:56.080619 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1427,"thread_start_us":1275,"threads_started":1}
I20260926 16:35:56.086989 26541 tablet_service.cc:1467] Processing CreateTablet for tablet 1d8c83597d814e78a5d8a1223036c8f7 (DEFAULT_TABLE table=test-table [id=b26d48bdc8c14635bf5f992a6a5dbed2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:35:56.088686 26541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d8c83597d814e78a5d8a1223036c8f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:35:56.090106 26618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:56.067389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:46097 (local address 127.0.0.1:38538)
0926 16:35:56.081624 (+ 14235us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:56.081648 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 16:35:56.081774 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:35:56.082800 (+ 1026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:35:56.082805 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:35:56.082828 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:35:56.083010 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:56.083025 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:56.086116 (+ 3091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:56.086127 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:56.087589 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:56.087602 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:56.089305 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:56.089311 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:56.089482 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:56.089514 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 16:35:56.089602 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 16:35:56.089746 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14064,"mutex_wait_us":102,"thread_start_us":18498,"threads_started":1}
I20260926 16:35:56.090154 26621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:56.068046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38538 (local address 127.6.33.1:46097)
0926 16:35:56.081867 (+ 13821us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:56.081881 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:56.081921 (+ 40us) server_negotiation.cc:407] Connection header received
0926 16:35:56.082049 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:56.082062 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:56.082377 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:56.083194 (+ 817us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:35:56.083526 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:56.085591 (+ 2065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:56.087751 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:56.089145 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:56.089296 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:56.089563 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:56.089638 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 16:35:56.089808 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13562,"thread_start_us":14461,"threads_started":1}
I20260926 16:35:56.091327 26622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:56.068781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.2:43853 (local address 127.0.0.1:44516)
0926 16:35:56.082776 (+ 13995us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:56.082791 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 16:35:56.082910 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:35:56.083814 (+ 904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:35:56.083819 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:35:56.083848 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:35:56.084295 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:56.084307 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:56.086181 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:56.086186 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:56.088370 (+ 2184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:56.088384 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:56.090494 (+ 2110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:56.090502 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:56.090682 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:56.090722 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 16:35:56.090819 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 16:35:56.090957 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13876,"mutex_wait_us":35,"thread_start_us":13736,"threads_started":1}
I20260926 16:35:56.091383 26623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:56.069301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44516 (local address 127.6.33.2:43853)
0926 16:35:56.082347 (+ 13046us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:56.082360 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:56.083064 (+ 704us) server_negotiation.cc:407] Connection header received
0926 16:35:56.083169 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:56.083177 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:56.083403 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:56.083584 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:35:56.084492 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:56.086014 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:56.088608 (+ 2594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:56.090016 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:56.090152 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:56.090369 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:56.090886 (+ 517us) server_negotiation.cc:299] Negotiation successful
0926 16:35:56.091051 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12867,"thread_start_us":12731,"threads_started":1}
I20260926 16:35:56.094290 26280 tablet_service.cc:1467] Processing CreateTablet for tablet 1d8c83597d814e78a5d8a1223036c8f7 (DEFAULT_TABLE table=test-table [id=b26d48bdc8c14635bf5f992a6a5dbed2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:35:56.095081 26411 tablet_service.cc:1467] Processing CreateTablet for tablet 1d8c83597d814e78a5d8a1223036c8f7 (DEFAULT_TABLE table=test-table [id=b26d48bdc8c14635bf5f992a6a5dbed2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:35:56.095831 26280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d8c83597d814e78a5d8a1223036c8f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:35:56.096369 26411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d8c83597d814e78a5d8a1223036c8f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:35:56.102187 26624 tablet_bootstrap.cc:492] T 1d8c83597d814e78a5d8a1223036c8f7 P 907182b2fba4422ebfa5ad9065af8c4a: Bootstrap starting.
I20260926 16:35:56.106262 26624 tablet_bootstrap.cc:654] T 1d8c83597d814e78a5d8a1223036c8f7 P 907182b2fba4422ebfa5ad9065af8c4a: Neither blocks nor log segments found. Creating new log.
I20260926 16:35:56.107306 26624 log.cc:824] T 1d8c83597d814e78a5d8a1223036c8f7 P 907182b2fba4422ebfa5ad9065af8c4a: Log is configured to *not* fsync() on all Append() calls
I20260926 16:35:56.110160 26624 tablet_bootstrap.cc:492] T 1d8c83597d814e78a5d8a1223036c8f7 P 907182b2fba4422ebfa5ad9065af8c4a: No bootstrap required, opened a new log
I20260926 16:35:56.110435 26624 ts_tablet_manager.cc:1397] T 1d8c83597d814e78a5d8a1223036c8f7 P 907182b2fba4422ebfa5ad9065af8c4a: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260926 16:35:56.111860 26626 tablet_bootstrap.cc:492] T 1d8c83597d814e78a5d8a1223036c8f7 P e6b4bfa6b3c94d8db962bbe5d44cf8b7: Bootstrap starting.
I20260926 16:35:56.112663 26627 tablet_bootstrap.cc:492] T 1d8c83597d814e78a5d8a1223036c8f7 P 47ee40ef8df74033b84845a19217ad43: Bootstrap starting.
I20260926 16:35:56.116026 26626 tablet_bootstrap.cc:654] T 1d8c83597d814e78a5d8a1223036c8f7 P e6b4bfa6b3c94d8db962bbe5d44cf8b7: Neither blocks nor log segments found. Creating new log.
I20260926 16:35:56.116317 26627 tablet_bootstrap.cc:654] T 1d8c83597d814e78a5d8a1223036c8f7 P 47ee40ef8df74033b84845a19217ad43: Neither blocks nor log segments found. Creating new log.
I20260926 16:35:56.117182 26626 log.cc:824] T 1d8c83597d814e78a5d8a1223036c8f7 P e6b4bfa6b3c94d8db962bbe5d44cf8b7: Log is configured to *not* fsync() on all Append() calls
I20260926 16:35:56.117532 26627 log.cc:824] T 1d8c83597d814e78a5d8a1223036c8f7 P 47ee40ef8df74033b84845a19217ad43: Log is configured to *not* fsync() on all Append() calls
I20260926 16:35:56.120168 26627 tablet_bootstrap.cc:492] T 1d8c83597d814e78a5d8a1223036c8f7 P 47ee40ef8df74033b84845a19217ad43: No bootstrap required, opened a new log
I20260926 16:35:56.120393 26626 tablet_bootstrap.cc:492] T 1d8c83597d814e78a5d8a1223036c8f7 P e6b4bfa6b3c94d8db962bbe5d44cf8b7: No bootstrap required, opened a new log
I20260926 16:35:56.120468 26627 ts_tablet_manager.cc:1397] T 1d8c83597d814e78a5d8a1223036c8f7 P 47ee40ef8df74033b84845a19217ad43: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260926 16:35:56.120669 26626 ts_tablet_manager.cc:1397] T 1d8c83597d814e78a5d8a1223036c8f7 P e6b4bfa6b3c94d8db962bbe5d44cf8b7: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.005s
I20260926 16:35:56.121249 26624 raft_consensus.cc:348] T 1d8c83597d814e78a5d8a1223036c8f7 P 907182b2fba4422ebfa5ad9065af8c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "907182b2fba4422ebfa5ad9065af8c4a" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 34025 } } peers { permanent_uuid: "e6b4bfa6b3c94d8db962bbe5d44cf8b7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 43853 } } peers { permanent_uuid: "47ee40ef8df74033b84845a19217ad43" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 46097 } }
I20260926 16:35:56.121712 26624 raft_consensus.cc:374] T 1d8c83597d814e78a5d8a1223036c8f7 P 907182b2fba4422ebfa5ad9065af8c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:35:56.121821 26624 raft_consensus.cc:729] T 1d8c83597d814e78a5d8a1223036c8f7 P 907182b2fba4422ebfa5ad9065af8c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 907182b2fba4422ebfa5ad9065af8c4a, State: Initialized, Role: FOLLOWER
I20260926 16:35:56.122476 26624 consensus_queue.cc:260] T 1d8c83597d814e78a5d8a1223036c8f7 P 907182b2fba4422ebfa5ad9065af8c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "907182b2fba4422ebfa5ad9065af8c4a" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 34025 } } peers { permanent_uuid: "e6b4bfa6b3c94d8db962bbe5d44cf8b7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 43853 } } peers { permanent_uuid: "47ee40ef8df74033b84845a19217ad43" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 46097 } }
I20260926 16:35:56.123711 26606 heartbeater.cc:503] Master 127.6.33.62:39461 was elected leader, sending a full tablet report...
I20260926 16:35:56.124428 26624 ts_tablet_manager.cc:1428] T 1d8c83597d814e78a5d8a1223036c8f7 P 907182b2fba4422ebfa5ad9065af8c4a: Time spent starting tablet: real 0.014s user 0.009s sys 0.005s
I20260926 16:35:56.129567 26627 raft_consensus.cc:348] T 1d8c83597d814e78a5d8a1223036c8f7 P 47ee40ef8df74033b84845a19217ad43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "907182b2fba4422ebfa5ad9065af8c4a" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 34025 } } peers { permanent_uuid: "e6b4bfa6b3c94d8db962bbe5d44cf8b7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 43853 } } peers { permanent_uuid: "47ee40ef8df74033b84845a19217ad43" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 46097 } }
I20260926 16:35:56.129937 26627 raft_consensus.cc:374] T 1d8c83597d814e78a5d8a1223036c8f7 P 47ee40ef8df74033b84845a19217ad43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:35:56.130056 26627 raft_consensus.cc:729] T 1d8c83597d814e78a5d8a1223036c8f7 P 47ee40ef8df74033b84845a19217ad43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47ee40ef8df74033b84845a19217ad43, State: Initialized, Role: FOLLOWER
I20260926 16:35:56.130662 26627 consensus_queue.cc:260] T 1d8c83597d814e78a5d8a1223036c8f7 P 47ee40ef8df74033b84845a19217ad43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "907182b2fba4422ebfa5ad9065af8c4a" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 34025 } } peers { permanent_uuid: "e6b4bfa6b3c94d8db962bbe5d44cf8b7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 43853 } } peers { permanent_uuid: "47ee40ef8df74033b84845a19217ad43" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 46097 } }
I20260926 16:35:56.130905 26626 raft_consensus.cc:348] T 1d8c83597d814e78a5d8a1223036c8f7 P e6b4bfa6b3c94d8db962bbe5d44cf8b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "907182b2fba4422ebfa5ad9065af8c4a" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 34025 } } peers { permanent_uuid: "e6b4bfa6b3c94d8db962bbe5d44cf8b7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 43853 } } peers { permanent_uuid: "47ee40ef8df74033b84845a19217ad43" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 46097 } }
I20260926 16:35:56.131354 26626 raft_consensus.cc:374] T 1d8c83597d814e78a5d8a1223036c8f7 P e6b4bfa6b3c94d8db962bbe5d44cf8b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:35:56.131454 26626 raft_consensus.cc:729] T 1d8c83597d814e78a5d8a1223036c8f7 P e6b4bfa6b3c94d8db962bbe5d44cf8b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6b4bfa6b3c94d8db962bbe5d44cf8b7, State: Initialized, Role: FOLLOWER
I20260926 16:35:56.132020 26626 consensus_queue.cc:260] T 1d8c83597d814e78a5d8a1223036c8f7 P e6b4bfa6b3c94d8db962bbe5d44cf8b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "907182b2fba4422ebfa5ad9065af8c4a" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 34025 } } peers { permanent_uuid: "e6b4bfa6b3c94d8db962bbe5d44cf8b7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 43853 } } peers { permanent_uuid: "47ee40ef8df74033b84845a19217ad43" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 46097 } }
I20260926 16:35:56.132426 26627 ts_tablet_manager.cc:1428] T 1d8c83597d814e78a5d8a1223036c8f7 P 47ee40ef8df74033b84845a19217ad43: Time spent starting tablet: real 0.012s user 0.008s sys 0.005s
I20260926 16:35:56.133028 26476 heartbeater.cc:503] Master 127.6.33.62:39461 was elected leader, sending a full tablet report...
I20260926 16:35:56.133644 26626 ts_tablet_manager.cc:1428] T 1d8c83597d814e78a5d8a1223036c8f7 P e6b4bfa6b3c94d8db962bbe5d44cf8b7: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
W20260926 16:35:56.241734 26607 tablet.cc:2369] T 1d8c83597d814e78a5d8a1223036c8f7 P 907182b2fba4422ebfa5ad9065af8c4a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 16:35:56.275300 26477 tablet.cc:2369] T 1d8c83597d814e78a5d8a1223036c8f7 P e6b4bfa6b3c94d8db962bbe5d44cf8b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:35:56.288388 26631 raft_consensus.cc:482] T 1d8c83597d814e78a5d8a1223036c8f7 P 47ee40ef8df74033b84845a19217ad43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:35:56.288725 26631 raft_consensus.cc:504] T 1d8c83597d814e78a5d8a1223036c8f7 P 47ee40ef8df74033b84845a19217ad43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "907182b2fba4422ebfa5ad9065af8c4a" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 34025 } } peers { permanent_uuid: "e6b4bfa6b3c94d8db962bbe5d44cf8b7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 43853 } } peers { permanent_uuid: "47ee40ef8df74033b84845a19217ad43" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 46097 } }
I20260926 16:35:56.290534 26631 leader_election.cc:288] T 1d8c83597d814e78a5d8a1223036c8f7 P 47ee40ef8df74033b84845a19217ad43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 907182b2fba4422ebfa5ad9065af8c4a (127.6.33.3:34025), e6b4bfa6b3c94d8db962bbe5d44cf8b7 (127.6.33.2:43853)
I20260926 16:35:56.297379 26619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:56.290962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:51035 (local address 127.6.33.3:34025)
0926 16:35:56.291175 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:56.291179 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:56.291832 (+ 653us) server_negotiation.cc:407] Connection header received
0926 16:35:56.292115 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:56.292121 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:56.292207 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:56.292341 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:35:56.292973 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:56.293697 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:56.295590 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:56.296438 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:56.296570 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:56.296683 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:56.296999 (+ 316us) server_negotiation.cc:299] Negotiation successful
0926 16:35:56.297130 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260926 16:35:56.297500 26633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:56.290979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.3:34025 (local address 127.6.33.1:51035)
0926 16:35:56.291680 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:56.291701 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 16:35:56.291848 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:35:56.292364 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:35:56.292373 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:35:56.292401 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:35:56.292833 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:56.292843 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:56.293824 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:56.293831 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:56.295376 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:56.295388 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:56.296615 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:56.296621 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:56.296830 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:56.296864 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 16:35:56.296972 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 16:35:56.297107 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":376,"threads_started":1}
I20260926 16:35:56.298113 26623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:56.292063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.33.1:38457 (local address 127.6.33.2:43853)
0926 16:35:56.292242 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:56.292249 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:56.292642 (+ 393us) server_negotiation.cc:407] Connection header received
0926 16:35:56.292868 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:56.292872 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:56.292944 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:56.293039 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:35:56.293557 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:56.294232 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:56.296148 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:56.297040 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:56.297172 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:56.297281 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:56.297728 (+ 447us) server_negotiation.cc:299] Negotiation successful
0926 16:35:56.297867 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 16:35:56.298151 26634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:56.291852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.2:43853 (local address 127.6.33.1:38457)
0926 16:35:56.292527 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:56.292541 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 16:35:56.292693 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:35:56.293053 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:35:56.293057 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:35:56.293079 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:35:56.293395 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:56.293403 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:56.294390 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:56.294397 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:56.295963 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:35:56.295975 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:35:56.297270 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:35:56.297278 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:35:56.297482 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:56.297525 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 16:35:56.297673 (+ 148us) client_negotiation.cc:240] Negotiation successful
0926 16:35:56.297788 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":368,"threads_started":1}
I20260926 16:35:56.298641 26561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d8c83597d814e78a5d8a1223036c8f7" candidate_uuid: "47ee40ef8df74033b84845a19217ad43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "907182b2fba4422ebfa5ad9065af8c4a" is_pre_election: true
I20260926 16:35:56.299139 26561 raft_consensus.cc:2454] T 1d8c83597d814e78a5d8a1223036c8f7 P 907182b2fba4422ebfa5ad9065af8c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47ee40ef8df74033b84845a19217ad43 in term 0.
I20260926 16:35:56.299115 26431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d8c83597d814e78a5d8a1223036c8f7" candidate_uuid: "47ee40ef8df74033b84845a19217ad43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6b4bfa6b3c94d8db962bbe5d44cf8b7" is_pre_election: true
I20260926 16:35:56.299674 26431 raft_consensus.cc:2454] T 1d8c83597d814e78a5d8a1223036c8f7 P e6b4bfa6b3c94d8db962bbe5d44cf8b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47ee40ef8df74033b84845a19217ad43 in term 0.
I20260926 16:35:56.299826 26236 leader_election.cc:302] T 1d8c83597d814e78a5d8a1223036c8f7 P 47ee40ef8df74033b84845a19217ad43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47ee40ef8df74033b84845a19217ad43, 907182b2fba4422ebfa5ad9065af8c4a; no voters:
I20260926 16:35:56.300236 26631 raft_consensus.cc:2781] T 1d8c83597d814e78a5d8a1223036c8f7 P 47ee40ef8df74033b84845a19217ad43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:35:56.300413 26631 raft_consensus.cc:482] T 1d8c83597d814e78a5d8a1223036c8f7 P 47ee40ef8df74033b84845a19217ad43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:35:56.300532 26631 raft_consensus.cc:3037] T 1d8c83597d814e78a5d8a1223036c8f7 P 47ee40ef8df74033b84845a19217ad43 [term 0 FOLLOWER]: Advancing to term 1
W20260926 16:35:56.300625 26346 tablet.cc:2369] T 1d8c83597d814e78a5d8a1223036c8f7 P 47ee40ef8df74033b84845a19217ad43: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:35:56.303084 26631 raft_consensus.cc:504] T 1d8c83597d814e78a5d8a1223036c8f7 P 47ee40ef8df74033b84845a19217ad43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "907182b2fba4422ebfa5ad9065af8c4a" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 34025 } } peers { permanent_uuid: "e6b4bfa6b3c94d8db962bbe5d44cf8b7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 43853 } } peers { permanent_uuid: "47ee40ef8df74033b84845a19217ad43" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 46097 } }
I20260926 16:35:56.303722 26631 leader_election.cc:288] T 1d8c83597d814e78a5d8a1223036c8f7 P 47ee40ef8df74033b84845a19217ad43 [CANDIDATE]: Term 1 election: Requested vote from peers 907182b2fba4422ebfa5ad9065af8c4a (127.6.33.3:34025), e6b4bfa6b3c94d8db962bbe5d44cf8b7 (127.6.33.2:43853)
I20260926 16:35:56.304106 26561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d8c83597d814e78a5d8a1223036c8f7" candidate_uuid: "47ee40ef8df74033b84845a19217ad43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "907182b2fba4422ebfa5ad9065af8c4a"
I20260926 16:35:56.304193 26431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d8c83597d814e78a5d8a1223036c8f7" candidate_uuid: "47ee40ef8df74033b84845a19217ad43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6b4bfa6b3c94d8db962bbe5d44cf8b7"
I20260926 16:35:56.304344 26561 raft_consensus.cc:3037] T 1d8c83597d814e78a5d8a1223036c8f7 P 907182b2fba4422ebfa5ad9065af8c4a [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:35:56.304410 26431 raft_consensus.cc:3037] T 1d8c83597d814e78a5d8a1223036c8f7 P e6b4bfa6b3c94d8db962bbe5d44cf8b7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:35:56.306664 26561 raft_consensus.cc:2454] T 1d8c83597d814e78a5d8a1223036c8f7 P 907182b2fba4422ebfa5ad9065af8c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47ee40ef8df74033b84845a19217ad43 in term 1.
I20260926 16:35:56.306824 26431 raft_consensus.cc:2454] T 1d8c83597d814e78a5d8a1223036c8f7 P e6b4bfa6b3c94d8db962bbe5d44cf8b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47ee40ef8df74033b84845a19217ad43 in term 1.
I20260926 16:35:56.307070 26236 leader_election.cc:302] T 1d8c83597d814e78a5d8a1223036c8f7 P 47ee40ef8df74033b84845a19217ad43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47ee40ef8df74033b84845a19217ad43, 907182b2fba4422ebfa5ad9065af8c4a; no voters:
I20260926 16:35:56.307343 26631 raft_consensus.cc:2781] T 1d8c83597d814e78a5d8a1223036c8f7 P 47ee40ef8df74033b84845a19217ad43 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:35:56.307818 26631 raft_consensus.cc:686] T 1d8c83597d814e78a5d8a1223036c8f7 P 47ee40ef8df74033b84845a19217ad43 [term 1 LEADER]: Becoming Leader. State: Replica: 47ee40ef8df74033b84845a19217ad43, State: Running, Role: LEADER
I20260926 16:35:56.308279 26631 consensus_queue.cc:237] T 1d8c83597d814e78a5d8a1223036c8f7 P 47ee40ef8df74033b84845a19217ad43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "907182b2fba4422ebfa5ad9065af8c4a" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 34025 } } peers { permanent_uuid: "e6b4bfa6b3c94d8db962bbe5d44cf8b7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 43853 } } peers { permanent_uuid: "47ee40ef8df74033b84845a19217ad43" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 46097 } }
I20260926 16:35:56.312690 26159 catalog_manager.cc:5495] T 1d8c83597d814e78a5d8a1223036c8f7 P 47ee40ef8df74033b84845a19217ad43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 47ee40ef8df74033b84845a19217ad43 (127.6.33.1). New cstate: current_term: 1 leader_uuid: "47ee40ef8df74033b84845a19217ad43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "907182b2fba4422ebfa5ad9065af8c4a" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 34025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6b4bfa6b3c94d8db962bbe5d44cf8b7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 43853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47ee40ef8df74033b84845a19217ad43" member_type: VOTER last_known_addr { host: "127.6.33.1" port: 46097 } health_report { overall_health: HEALTHY } } }
I20260926 16:35:56.371246 26621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:56.365928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37288 (local address 127.6.33.1:46097)
0926 16:35:56.366180 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:56.366187 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:56.366205 (+ 18us) server_negotiation.cc:407] Connection header received
0926 16:35:56.366422 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:56.366426 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:56.366537 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:56.366682 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:35:56.367048 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:56.367715 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:56.369189 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:56.369596 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:56.369729 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:56.370187 (+ 458us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:35:56.370691 (+ 504us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:35:56.370851 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:56.370906 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 16:35:56.370985 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 16:35:56.379479 26561 raft_consensus.cc:1261] T 1d8c83597d814e78a5d8a1223036c8f7 P 907182b2fba4422ebfa5ad9065af8c4a [term 1 FOLLOWER]: Refusing update from remote peer 47ee40ef8df74033b84845a19217ad43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:35:56.379488 26431 raft_consensus.cc:1261] T 1d8c83597d814e78a5d8a1223036c8f7 P e6b4bfa6b3c94d8db962bbe5d44cf8b7 [term 1 FOLLOWER]: Refusing update from remote peer 47ee40ef8df74033b84845a19217ad43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:35:56.380384 26635 consensus_queue.cc:1035] T 1d8c83597d814e78a5d8a1223036c8f7 P 47ee40ef8df74033b84845a19217ad43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6b4bfa6b3c94d8db962bbe5d44cf8b7" member_type: VOTER last_known_addr { host: "127.6.33.2" port: 43853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:35:56.380867 26631 consensus_queue.cc:1035] T 1d8c83597d814e78a5d8a1223036c8f7 P 47ee40ef8df74033b84845a19217ad43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "907182b2fba4422ebfa5ad9065af8c4a" member_type: VOTER last_known_addr { host: "127.6.33.3" port: 34025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:35:56.396087 26639 mvcc.cc:205] Tried to move back new op lower bound from 7333644518920892416 to 7333644518642470912. Current Snapshot: MvccSnapshot[applied={T|T < 7333644518920892416}]
I20260926 16:35:56.404506 26158 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:35:56.404767 26158 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:35:56.408519 26158 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1ddb40f5d77347d091334b51691ec041: Sending DeleteTablet for 3 replicas of tablet 1d8c83597d814e78a5d8a1223036c8f7
I20260926 16:35:56.409708 26541 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d8c83597d814e78a5d8a1223036c8f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:35:56 UTC) from {username='slave'} at 127.0.0.1:58128
I20260926 16:35:56.410137 26411 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d8c83597d814e78a5d8a1223036c8f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:35:56 UTC) from {username='slave'} at 127.0.0.1:44516
I20260926 16:35:56.410141 26280 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d8c83597d814e78a5d8a1223036c8f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:35:56 UTC) from {username='slave'} at 127.0.0.1:38538
I20260926 16:35:56.411813 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 26218
I20260926 16:35:56.446099 26647 tablet_replica.cc:330] T 1d8c83597d814e78a5d8a1223036c8f7 P 907182b2fba4422ebfa5ad9065af8c4a: stopping tablet replica
I20260926 16:35:56.446800 26647 raft_consensus.cc:2229] T 1d8c83597d814e78a5d8a1223036c8f7 P 907182b2fba4422ebfa5ad9065af8c4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:35:56.447984 26648 tablet_replica.cc:330] T 1d8c83597d814e78a5d8a1223036c8f7 P e6b4bfa6b3c94d8db962bbe5d44cf8b7: stopping tablet replica
I20260926 16:35:56.448405 26648 raft_consensus.cc:2229] T 1d8c83597d814e78a5d8a1223036c8f7 P e6b4bfa6b3c94d8db962bbe5d44cf8b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:35:56.449733 26648 raft_consensus.cc:2258] T 1d8c83597d814e78a5d8a1223036c8f7 P e6b4bfa6b3c94d8db962bbe5d44cf8b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:35:56.450274 26647 raft_consensus.cc:2258] T 1d8c83597d814e78a5d8a1223036c8f7 P 907182b2fba4422ebfa5ad9065af8c4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:35:56.453560 26647 ts_tablet_manager.cc:1905] T 1d8c83597d814e78a5d8a1223036c8f7 P 907182b2fba4422ebfa5ad9065af8c4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:35:56.453709 26648 ts_tablet_manager.cc:1905] T 1d8c83597d814e78a5d8a1223036c8f7 P e6b4bfa6b3c94d8db962bbe5d44cf8b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:35:56.459267 26621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:56.413373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37304 (local address 127.6.33.1:46097)
0926 16:35:56.413601 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:56.413611 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:56.437688 (+ 24077us) server_negotiation.cc:407] Connection header received
0926 16:35:56.438484 (+ 796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:56.438495 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:56.438635 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:56.438744 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:35:56.439959 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:56.442005 (+ 2046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:56.449488 (+ 7483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:56.450271 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:56.450429 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:56.451935 (+ 1506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:35:56.451954 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:35:56.451977 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:35:56.457641 (+ 5664us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:35:56.457711 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:35:56.457742 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:35:56.457866 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:56.458136 (+ 270us) server_negotiation.cc:299] Negotiation successful
0926 16:35:56.458271 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 16:35:56.460124 26340 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:35:56.464892 26647 ts_tablet_manager.cc:1918] T 1d8c83597d814e78a5d8a1223036c8f7 P 907182b2fba4422ebfa5ad9065af8c4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:35:56.465092 26647 log.cc:1196] T 1d8c83597d814e78a5d8a1223036c8f7 P 907182b2fba4422ebfa5ad9065af8c4a: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/ts-2/wal/wals/1d8c83597d814e78a5d8a1223036c8f7
I20260926 16:35:56.465611 26647 ts_tablet_manager.cc:1939] T 1d8c83597d814e78a5d8a1223036c8f7 P 907182b2fba4422ebfa5ad9065af8c4a: Deleting consensus metadata
I20260926 16:35:56.466753 26648 ts_tablet_manager.cc:1918] T 1d8c83597d814e78a5d8a1223036c8f7 P e6b4bfa6b3c94d8db962bbe5d44cf8b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:35:56.466935 26648 log.cc:1196] T 1d8c83597d814e78a5d8a1223036c8f7 P e6b4bfa6b3c94d8db962bbe5d44cf8b7: Deleting WAL directory at /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440233850748-6276-0/minicluster-data/ts-1/wal/wals/1d8c83597d814e78a5d8a1223036c8f7
I20260926 16:35:56.467494 26648 ts_tablet_manager.cc:1939] T 1d8c83597d814e78a5d8a1223036c8f7 P e6b4bfa6b3c94d8db962bbe5d44cf8b7: Deleting consensus metadata
I20260926 16:35:56.467603 26146 catalog_manager.cc:4841] TS 907182b2fba4422ebfa5ad9065af8c4a (127.6.33.3:34025): tablet 1d8c83597d814e78a5d8a1223036c8f7 (table test-table [id=b26d48bdc8c14635bf5f992a6a5dbed2]) successfully deleted
I20260926 16:35:56.469434 26146 catalog_manager.cc:4841] TS e6b4bfa6b3c94d8db962bbe5d44cf8b7 (127.6.33.2:43853): tablet 1d8c83597d814e78a5d8a1223036c8f7 (table test-table [id=b26d48bdc8c14635bf5f992a6a5dbed2]) successfully deleted
I20260926 16:35:56.559866 26649 tablet_replica.cc:330] T 1d8c83597d814e78a5d8a1223036c8f7 P 47ee40ef8df74033b84845a19217ad43: stopping tablet replica
I20260926 16:35:56.560570 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 26218
I20260926 16:35:56.579986 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 26349
W20260926 16:35:56.579959 26143 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.33.1:46097 (error 108)
I20260926 16:35:56.581831 26622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:56.581314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:46097 (local address 127.0.0.1:37316)
0926 16:35:56.581507 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:56.581635 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.1:46097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260926 16:35:56.582201 26143 catalog_manager.cc:4584] TS 47ee40ef8df74033b84845a19217ad43 (127.6.33.1:46097): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1d8c83597d814e78a5d8a1223036c8f7: Network error: Client connection negotiation failed: client connection to 127.6.33.1:46097: connect: Connection refused (error 111)
I20260926 16:35:56.587122 26623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:56.580844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52716 (local address 127.6.33.2:43853)
0926 16:35:56.581031 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:56.581038 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:56.581061 (+ 23us) server_negotiation.cc:407] Connection header received
0926 16:35:56.581262 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:56.581267 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:56.581371 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:56.581506 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:35:56.581894 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:56.582776 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:56.584079 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:56.584515 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:56.584620 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:56.585692 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:35:56.585700 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:35:56.585714 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:35:56.586357 (+ 643us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:35:56.586406 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:35:56.586426 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:35:56.586527 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:56.586707 (+ 180us) server_negotiation.cc:299] Negotiation successful
0926 16:35:56.586805 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 16:35:56.587833 26471 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:35:56.636102 26622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:56.635531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:46097 (local address 127.0.0.1:37322)
0926 16:35:56.635770 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:56.635868 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.1:46097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260926 16:35:56.637270 26622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:56.636888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:46097 (local address 127.0.0.1:37334)
0926 16:35:56.637050 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:56.637143 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.1:46097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 16:35:56.677906 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 26349
I20260926 16:35:56.695077 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 26479
I20260926 16:35:56.697749 26622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:56.697244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:46097 (local address 127.0.0.1:37338)
0926 16:35:56.697442 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:56.697555 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.1:46097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260926 16:35:56.698755 26622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:56.698430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:46097 (local address 127.0.0.1:37342)
0926 16:35:56.698572 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:56.698645 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.1:46097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 16:35:56.702081 26619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:56.695817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33980 (local address 127.6.33.3:34025)
0926 16:35:56.696031 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:56.696038 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:56.696116 (+ 78us) server_negotiation.cc:407] Connection header received
0926 16:35:56.696178 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:56.696183 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:56.696281 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:56.696365 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:35:56.696911 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:56.697730 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:56.699132 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:56.699508 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:56.699640 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:56.700663 (+ 1023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:35:56.700672 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:35:56.700687 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:35:56.701357 (+ 670us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:35:56.701404 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:35:56.701428 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:35:56.701547 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:56.701683 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 16:35:56.701788 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 16:35:56.702740 26601 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:35:56.779397 26622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:56.778794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:46097 (local address 127.0.0.1:37354)
0926 16:35:56.779042 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:56.779164 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.1:46097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260926 16:35:56.780637 26622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:56.780170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.1:46097 (local address 127.0.0.1:37362)
0926 16:35:56.780388 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0926 16:35:56.780496 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.1:46097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260926 16:35:56.794348 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 26479
I20260926 16:35:56.811034 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 26128
I20260926 16:35:56.812171 26189 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:35:56.922156 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 26128
2026-09-26T16:35:56Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2989 ms)
[----------] 6 tests from EncryptionPolicyTest (22885 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440233850748-6276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:35:57 dist-test-slave-khd3 krb5kdc[26658](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:35:57 dist-test-slave-khd3 krb5kdc[26658](info): set up 2 sockets
Sep 26 16:35:57 dist-test-slave-khd3 krb5kdc[26658](info): commencing operation
krb5kdc: starting...
W20260926 16:35:58.038172 6276 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Sep 26 16:35:58 dist-test-slave-khd3 krb5kdc[26658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:35:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:35:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440233850748-6276-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440233850748-6276-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 16:35:58.299712 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440233850748-6276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440233850748-6276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440233850748-6276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37557
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440233850748-6276-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37557
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:35:58.630721 26671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:35:58.631053 26671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:35:58.631206 26671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:35:58.638414 26671 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:35:58.638566 26671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:35:58.638669 26671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:35:58.638773 26671 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:35:58.638852 26671 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:35:58.648608 26671 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440233850748-6276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440233850748-6276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37557
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440233850748-6276-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440233850748-6276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:35:58.650039 26671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:35:58.651489 26671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:35:58.663817 26677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:35:58.663894 26679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:35:58.663776 26676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:35:58.664999 26671 server_base.cc:1034] running on GCE node
I20260926 16:35:58.665848 26671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:35:58.667460 26671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:35:58.668731 26671 hybrid_clock.cc:648] HybridClock initialized: now 1790440558668681 us; error 60 us; skew 500 ppm
I20260926 16:35:58.669260 26671 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:35:58 dist-test-slave-khd3 krb5kdc[26658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440558, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:35:58.672576 26671 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 16:35:58.674320 26671 webserver.cc:460] Webserver started at http://127.0.0.1:42621/ using document root <none> and password file <none>
I20260926 16:35:58.675062 26671 fs_manager.cc:359] Metadata directory not provided
I20260926 16:35:58.675206 26671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:35:58.675509 26671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:35:58.678690 26671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440233850748-6276-0/minicluster-data/master-0/data/instance:
uuid: "76aa9bf7a11140afb88233ad16688d1d"
format_stamp: "Formatted at 2026-09-26 16:35:58 on dist-test-slave-khd3"
server_key: "1c83da8fbb5053ceacffab772fec034f"
server_key_iv: "1566cd7cda18c0ac8664f0e82c051ab4"
server_key_version: "encryptionkey@0"
I20260926 16:35:58.679463 26671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440233850748-6276-0/minicluster-data/master-0/wal/instance:
uuid: "76aa9bf7a11140afb88233ad16688d1d"
format_stamp: "Formatted at 2026-09-26 16:35:58 on dist-test-slave-khd3"
server_key: "1c83da8fbb5053ceacffab772fec034f"
server_key_iv: "1566cd7cda18c0ac8664f0e82c051ab4"
server_key_version: "encryptionkey@0"
I20260926 16:35:58.684933 26671 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 16:35:58.688941 26686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:35:58.690275 26671 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:35:58.690464 26671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440233850748-6276-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440233850748-6276-0/minicluster-data/master-0/wal
uuid: "76aa9bf7a11140afb88233ad16688d1d"
format_stamp: "Formatted at 2026-09-26 16:35:58 on dist-test-slave-khd3"
server_key: "1c83da8fbb5053ceacffab772fec034f"
server_key_iv: "1566cd7cda18c0ac8664f0e82c051ab4"
server_key_version: "encryptionkey@0"
I20260926 16:35:58.690680 26671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440233850748-6276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440233850748-6276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440233850748-6276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:35:58.712679 26671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:35:58.716948 26671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:35:58.717244 26671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:35:58.736048 26671 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37557
I20260926 16:35:58.736078 26737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37557 every 8 connection(s)
I20260926 16:35:58.737509 26671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
I20260926 16:35:58.741285 26738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:35:58.744962 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 26671
I20260926 16:35:58.745177 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440233850748-6276-0/minicluster-data/master-0/wal/instance
I20260926 16:35:58.745784 6276 external_mini_cluster.cc:1428] Setting key 36a9f0a5917a79e486d5815d05c62965
I20260926 16:35:58.754451 26738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76aa9bf7a11140afb88233ad16688d1d: Bootstrap starting.
I20260926 16:35:58.758092 26738 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76aa9bf7a11140afb88233ad16688d1d: Neither blocks nor log segments found. Creating new log.
I20260926 16:35:58.759176 26738 log.cc:824] T 00000000000000000000000000000000 P 76aa9bf7a11140afb88233ad16688d1d: Log is configured to *not* fsync() on all Append() calls
Sep 26 16:35:58 dist-test-slave-khd3 krb5kdc[26658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 16:35:58.762250 26738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76aa9bf7a11140afb88233ad16688d1d: No bootstrap required, opened a new log
I20260926 16:35:58.768925 26741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:58.747797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50639 (local address 127.0.0.1:37557)
0926 16:35:58.748575 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:58.748587 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:58.748629 (+ 42us) server_negotiation.cc:407] Connection header received
0926 16:35:58.750290 (+ 1661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:58.750314 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:58.750758 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:58.751087 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:35:58.752952 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:58.754246 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:58.756009 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:58.756720 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:58.756956 (+ 236us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:58.761251 (+ 4295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:35:58.761259 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:35:58.761277 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:35:58.761313 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:35:58.764481 (+ 3168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:35:58.765397 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:35:58.765404 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:35:58.765411 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:35:58.765588 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:35:58.766055 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:35:58.766059 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:35:58.766061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:35:58.766610 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:35:58.766983 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:58.767478 (+ 495us) server_negotiation.cc:299] Negotiation successful
0926 16:35:58.767791 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":364,"threads_started":1}
I20260926 16:35:58.770555 26738 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76aa9bf7a11140afb88233ad16688d1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76aa9bf7a11140afb88233ad16688d1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37557 } }
I20260926 16:35:58.770925 26738 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76aa9bf7a11140afb88233ad16688d1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:35:58.771044 26738 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76aa9bf7a11140afb88233ad16688d1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76aa9bf7a11140afb88233ad16688d1d, State: Initialized, Role: FOLLOWER
I20260926 16:35:58.771756 26738 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76aa9bf7a11140afb88233ad16688d1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76aa9bf7a11140afb88233ad16688d1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37557 } }
I20260926 16:35:58.772055 26738 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76aa9bf7a11140afb88233ad16688d1d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:35:58.772246 26738 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76aa9bf7a11140afb88233ad16688d1d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:35:58.772487 26738 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76aa9bf7a11140afb88233ad16688d1d [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:35:58.773211 6276 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:35:58.775910 26738 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76aa9bf7a11140afb88233ad16688d1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76aa9bf7a11140afb88233ad16688d1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37557 } }
I20260926 16:35:58.776576 26738 leader_election.cc:302] T 00000000000000000000000000000000 P 76aa9bf7a11140afb88233ad16688d1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76aa9bf7a11140afb88233ad16688d1d; no voters:
I20260926 16:35:58.777344 26738 leader_election.cc:288] T 00000000000000000000000000000000 P 76aa9bf7a11140afb88233ad16688d1d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:35:58.777515 26744 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76aa9bf7a11140afb88233ad16688d1d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:35:58.779814 26738 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76aa9bf7a11140afb88233ad16688d1d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:35:58.780196 26744 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76aa9bf7a11140afb88233ad16688d1d [term 1 LEADER]: Becoming Leader. State: Replica: 76aa9bf7a11140afb88233ad16688d1d, State: Running, Role: LEADER
I20260926 16:35:58.780882 26744 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76aa9bf7a11140afb88233ad16688d1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76aa9bf7a11140afb88233ad16688d1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37557 } }
I20260926 16:35:58.784142 26749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76aa9bf7a11140afb88233ad16688d1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76aa9bf7a11140afb88233ad16688d1d. Latest consensus state: current_term: 1 leader_uuid: "76aa9bf7a11140afb88233ad16688d1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76aa9bf7a11140afb88233ad16688d1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37557 } } }
I20260926 16:35:58.784546 26749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76aa9bf7a11140afb88233ad16688d1d [sys.catalog]: This master's current role is: LEADER
I20260926 16:35:58.785095 26748 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76aa9bf7a11140afb88233ad16688d1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76aa9bf7a11140afb88233ad16688d1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76aa9bf7a11140afb88233ad16688d1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37557 } } }
I20260926 16:35:58.785384 26748 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76aa9bf7a11140afb88233ad16688d1d [sys.catalog]: This master's current role is: LEADER
I20260926 16:35:58.802815 26741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:35:58.779384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40411 (local address 127.0.0.1:37557)
0926 16:35:58.779594 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 16:35:58.779604 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:35:58.780093 (+ 489us) server_negotiation.cc:407] Connection header received
0926 16:35:58.780362 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:35:58.780367 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:35:58.780455 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:35:58.780553 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:35:58.783199 (+ 2646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:58.784710 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:58.787234 (+ 2524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:35:58.788184 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:35:58.788303 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:35:58.792733 (+ 4430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:35:58.792744 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:35:58.792778 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:35:58.792833 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:35:58.797881 (+ 5048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:35:58.799269 (+ 1388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:35:58.799278 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:35:58.799284 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:35:58.799485 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:35:58.800190 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:35:58.800198 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:35:58.800203 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:35:58.800801 (+ 598us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:35:58.801127 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0926 16:35:58.802021 (+ 894us) server_negotiation.cc:299] Negotiation successful
0926 16:35:58.802273 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 16:35:58.808919 26758 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:35:58.813999 26758 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:35:58.822685 26758 catalog_manager.cc:1347] Generated new cluster ID: 44b2cfebca2a49efac81978f501786d3
I20260926 16:35:58.822809 26758 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:35:58.844866 26758 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:35:58.846827 26758 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:35:58.873517 26758 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76aa9bf7a11140afb88233ad16688d1d: Generated new TSK 0
I20260926 16:35:58.874469 26758 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:35:58.886510 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 26671
I20260926 16:35:58.888530 26733 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:35:58.983330 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 26671
W20260926 16:35:59.000808 26656 connection.cc:535] client connection to 127.0.0.1:37557 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37557): Connection reset by peer (error 104)
2026-09-26T16:35:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2051 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440233850748-6276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:35:59 dist-test-slave-khd3 krb5kdc[26771](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:35:59 dist-test-slave-khd3 krb5kdc[26771](info): set up 2 sockets
Sep 26 16:35:59 dist-test-slave-khd3 krb5kdc[26771](info): commencing operation
krb5kdc: starting...
W20260926 16:36:00.102432 6276 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Sep 26 16:36:00 dist-test-slave-khd3 krb5kdc[26771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:36:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:36:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440233850748-6276-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440233850748-6276-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 16:36:00.362314 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440233850748-6276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440233850748-6276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440233850748-6276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41033
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440233850748-6276-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41033
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:36:00.697016 26784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:36:00.697382 26784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:36:00.697542 26784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:36:00.705071 26784 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:36:00.705237 26784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:36:00.705344 26784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:36:00.705453 26784 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:36:00.705529 26784 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:36:00.715852 26784 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440233850748-6276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440233850748-6276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41033
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440233850748-6276-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440233850748-6276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:36:00.717280 26784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:36:00.718639 26784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:36:00.730799 26789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:36:00.730859 26790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:36:00.730927 26792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:36:00.731673 26784 server_base.cc:1034] running on GCE node
I20260926 16:36:00.732547 26784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:36:00.734249 26784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:36:00.735534 26784 hybrid_clock.cc:648] HybridClock initialized: now 1790440560735480 us; error 49 us; skew 500 ppm
I20260926 16:36:00.735917 26784 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:36:00 dist-test-slave-khd3 krb5kdc[26771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440560, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:36:00.739320 26784 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 16:36:00.741016 26784 webserver.cc:460] Webserver started at http://127.0.0.1:36321/ using document root <none> and password file <none>
I20260926 16:36:00.741739 26784 fs_manager.cc:359] Metadata directory not provided
I20260926 16:36:00.741880 26784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:36:00.742197 26784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:36:00.745393 26784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440233850748-6276-0/minicluster-data/master-0/data/instance:
uuid: "7c34f9503d7a4b798adf22abed59380d"
format_stamp: "Formatted at 2026-09-26 16:36:00 on dist-test-slave-khd3"
server_key: "18d9cb86dd95dcac4281fe9baaffe599"
server_key_iv: "03d11053e871a2bebf2cb77b257956ba"
server_key_version: "encryptionkey@0"
I20260926 16:36:00.746205 26784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440233850748-6276-0/minicluster-data/master-0/wal/instance:
uuid: "7c34f9503d7a4b798adf22abed59380d"
format_stamp: "Formatted at 2026-09-26 16:36:00 on dist-test-slave-khd3"
server_key: "18d9cb86dd95dcac4281fe9baaffe599"
server_key_iv: "03d11053e871a2bebf2cb77b257956ba"
server_key_version: "encryptionkey@0"
I20260926 16:36:00.751739 26784 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 16:36:00.755491 26799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:36:00.757067 26784 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:36:00.757287 26784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440233850748-6276-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440233850748-6276-0/minicluster-data/master-0/wal
uuid: "7c34f9503d7a4b798adf22abed59380d"
format_stamp: "Formatted at 2026-09-26 16:36:00 on dist-test-slave-khd3"
server_key: "18d9cb86dd95dcac4281fe9baaffe599"
server_key_iv: "03d11053e871a2bebf2cb77b257956ba"
server_key_version: "encryptionkey@0"
I20260926 16:36:00.757494 26784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440233850748-6276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440233850748-6276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440233850748-6276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:36:00.778151 26784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:36:00.782413 26784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:36:00.782717 26784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:36:00.801767 26784 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41033
I20260926 16:36:00.801812 26850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41033 every 8 connection(s)
I20260926 16:36:00.803201 26784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
I20260926 16:36:00.806998 26851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:36:00.807042 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 26784
I20260926 16:36:00.807226 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440233850748-6276-0/minicluster-data/master-0/wal/instance
I20260926 16:36:00.807782 6276 external_mini_cluster.cc:1428] Setting key 32f3e1acf7bff68668abd4b180d5cfb3
I20260926 16:36:00.821740 26851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c34f9503d7a4b798adf22abed59380d: Bootstrap starting.
I20260926 16:36:00.822234 26854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:00.809542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59655 (local address 127.0.0.1:41033)
0926 16:36:00.810443 (+ 901us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:00.810455 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:00.810497 (+ 42us) server_negotiation.cc:407] Connection header received
0926 16:36:00.812245 (+ 1748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:00.812269 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:00.812726 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:00.813193 (+ 467us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:00.813801 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:00.815423 (+ 1622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:00.817576 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:00.818298 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:00.818455 (+ 157us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:00.819825 (+ 1370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:00.819832 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:00.819847 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:36:00.820404 (+ 557us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:00.820480 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:36:00.820504 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:00.820622 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:00.820971 (+ 349us) server_negotiation.cc:299] Negotiation successful
0926 16:36:00.821194 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":686,"thread_start_us":456,"threads_started":1}
I20260926 16:36:00.825099 26851 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c34f9503d7a4b798adf22abed59380d: Neither blocks nor log segments found. Creating new log.
I20260926 16:36:00.826046 6276 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:36:00.826284 26851 log.cc:824] T 00000000000000000000000000000000 P 7c34f9503d7a4b798adf22abed59380d: Log is configured to *not* fsync() on all Append() calls
I20260926 16:36:00.829653 26851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c34f9503d7a4b798adf22abed59380d: No bootstrap required, opened a new log
I20260926 16:36:00.838444 26854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:00.831264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54609 (local address 127.0.0.1:41033)
0926 16:36:00.831467 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:00.831474 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:00.832022 (+ 548us) server_negotiation.cc:407] Connection header received
0926 16:36:00.832160 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:00.832163 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:00.832227 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:00.832323 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:00.832933 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:00.833754 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:00.835675 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:00.836154 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:00.836255 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:00.837638 (+ 1383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:00.837642 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:00.837646 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:36:00.837665 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:00.837707 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:36:00.837723 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:00.837814 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:00.838020 (+ 206us) server_negotiation.cc:299] Negotiation successful
0926 16:36:00.838070 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 16:36:00.840039 26851 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c34f9503d7a4b798adf22abed59380d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c34f9503d7a4b798adf22abed59380d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41033 } }
I20260926 16:36:00.840431 26851 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c34f9503d7a4b798adf22abed59380d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:36:00.840548 26851 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c34f9503d7a4b798adf22abed59380d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c34f9503d7a4b798adf22abed59380d, State: Initialized, Role: FOLLOWER
I20260926 16:36:00.841219 26851 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c34f9503d7a4b798adf22abed59380d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c34f9503d7a4b798adf22abed59380d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41033 } }
I20260926 16:36:00.841446 26851 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c34f9503d7a4b798adf22abed59380d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:36:00.841596 26851 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c34f9503d7a4b798adf22abed59380d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:36:00.841753 26851 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c34f9503d7a4b798adf22abed59380d [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:36:00.843987 26851 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c34f9503d7a4b798adf22abed59380d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c34f9503d7a4b798adf22abed59380d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41033 } }
I20260926 16:36:00.844501 26851 leader_election.cc:302] T 00000000000000000000000000000000 P 7c34f9503d7a4b798adf22abed59380d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c34f9503d7a4b798adf22abed59380d; no voters:
I20260926 16:36:00.845105 26851 leader_election.cc:288] T 00000000000000000000000000000000 P 7c34f9503d7a4b798adf22abed59380d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:36:00.845238 26861 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c34f9503d7a4b798adf22abed59380d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:36:00.846463 26861 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c34f9503d7a4b798adf22abed59380d [term 1 LEADER]: Becoming Leader. State: Replica: 7c34f9503d7a4b798adf22abed59380d, State: Running, Role: LEADER
I20260926 16:36:00.846975 26861 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c34f9503d7a4b798adf22abed59380d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c34f9503d7a4b798adf22abed59380d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41033 } }
I20260926 16:36:00.847289 26851 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c34f9503d7a4b798adf22abed59380d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:36:00.849918 26862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c34f9503d7a4b798adf22abed59380d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c34f9503d7a4b798adf22abed59380d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c34f9503d7a4b798adf22abed59380d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41033 } } }
I20260926 16:36:00.850030 26863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c34f9503d7a4b798adf22abed59380d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c34f9503d7a4b798adf22abed59380d. Latest consensus state: current_term: 1 leader_uuid: "7c34f9503d7a4b798adf22abed59380d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c34f9503d7a4b798adf22abed59380d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41033 } } }
I20260926 16:36:00.850311 26862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c34f9503d7a4b798adf22abed59380d [sys.catalog]: This master's current role is: LEADER
I20260926 16:36:00.850418 26863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c34f9503d7a4b798adf22abed59380d [sys.catalog]: This master's current role is: LEADER
I20260926 16:36:00.852731 26869 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:36:00.858129 26869 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:36:00.868431 26869 catalog_manager.cc:1347] Generated new cluster ID: a21e71bbe3b44bc19112863a98f481f6
I20260926 16:36:00.868556 26869 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:36:00.893824 26869 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:36:00.895146 26869 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:36:00.904500 26869 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c34f9503d7a4b798adf22abed59380d: Generated new TSK 0
I20260926 16:36:00.905375 26869 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:36:00.915913 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 26784
W20260926 16:36:00.917678 26846 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59655
W20260926 16:36:00.918509 6276 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 26784: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 16:36:00.918720 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 26784
W20260926 16:36:00.935793 26769 connection.cc:535] client connection to 127.0.0.1:41033 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41033): Connection reset by peer (error 104)
2026-09-26T16:36:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1934 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440233850748-6276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:36:01 dist-test-slave-khd3 krb5kdc[26882](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:36:01 dist-test-slave-khd3 krb5kdc[26882](info): set up 2 sockets
Sep 26 16:36:01 dist-test-slave-khd3 krb5kdc[26882](info): commencing operation
krb5kdc: starting...
W20260926 16:36:02.030448 6276 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Sep 26 16:36:02 dist-test-slave-khd3 krb5kdc[26882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:36:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:36:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440233850748-6276-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440233850748-6276-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 16:36:02.297217 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440233850748-6276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440233850748-6276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440233850748-6276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45133
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440233850748-6276-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45133
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:36:02.646066 26895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:36:02.646575 26895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:36:02.646756 26895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:36:02.656431 26895 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:36:02.656555 26895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:36:02.656653 26895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:36:02.656739 26895 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:36:02.656828 26895 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:36:02.667235 26895 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440233850748-6276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440233850748-6276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45133
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440233850748-6276-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440233850748-6276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:36:02.668596 26895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:36:02.670001 26895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:36:02.682220 26901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:36:02.682308 26903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:36:02.682260 26900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:36:02.683212 26895 server_base.cc:1034] running on GCE node
I20260926 16:36:02.684059 26895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:36:02.685648 26895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:36:02.686870 26895 hybrid_clock.cc:648] HybridClock initialized: now 1790440562686835 us; error 33 us; skew 500 ppm
I20260926 16:36:02.687256 26895 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:36:02 dist-test-slave-khd3 krb5kdc[26882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440562, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:36:02.690510 26895 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 16:36:02.692023 26895 webserver.cc:460] Webserver started at http://127.0.0.1:39673/ using document root <none> and password file <none>
I20260926 16:36:02.692728 26895 fs_manager.cc:359] Metadata directory not provided
I20260926 16:36:02.692901 26895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:36:02.693188 26895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:36:02.696295 26895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440233850748-6276-0/minicluster-data/master-0/data/instance:
uuid: "236020158c58472a97d740325ee47b76"
format_stamp: "Formatted at 2026-09-26 16:36:02 on dist-test-slave-khd3"
server_key: "b4b9b0d35836a24061e0e8db8e5d08ea"
server_key_iv: "84a33f106af70732cbffa9d64f39366a"
server_key_version: "encryptionkey@0"
I20260926 16:36:02.697171 26895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440233850748-6276-0/minicluster-data/master-0/wal/instance:
uuid: "236020158c58472a97d740325ee47b76"
format_stamp: "Formatted at 2026-09-26 16:36:02 on dist-test-slave-khd3"
server_key: "b4b9b0d35836a24061e0e8db8e5d08ea"
server_key_iv: "84a33f106af70732cbffa9d64f39366a"
server_key_version: "encryptionkey@0"
I20260926 16:36:02.702607 26895 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260926 16:36:02.706274 26910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:36:02.707724 26895 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:36:02.707907 26895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440233850748-6276-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440233850748-6276-0/minicluster-data/master-0/wal
uuid: "236020158c58472a97d740325ee47b76"
format_stamp: "Formatted at 2026-09-26 16:36:02 on dist-test-slave-khd3"
server_key: "b4b9b0d35836a24061e0e8db8e5d08ea"
server_key_iv: "84a33f106af70732cbffa9d64f39366a"
server_key_version: "encryptionkey@0"
I20260926 16:36:02.708122 26895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440233850748-6276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440233850748-6276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440233850748-6276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:36:02.714008 26895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:36:02.718660 26895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:36:02.718959 26895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:36:02.737393 26895 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45133
I20260926 16:36:02.737433 26961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45133 every 8 connection(s)
I20260926 16:36:02.738855 26895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
I20260926 16:36:02.742805 26962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:36:02.745679 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 26895
I20260926 16:36:02.745854 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440233850748-6276-0/minicluster-data/master-0/wal/instance
I20260926 16:36:02.746419 6276 external_mini_cluster.cc:1428] Setting key 9e939af9721c886a4bcac2f1a47722c0
I20260926 16:36:02.754071 26965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:02.748228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37449 (local address 127.0.0.1:45133)
0926 16:36:02.749040 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:02.749051 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:02.749098 (+ 47us) server_negotiation.cc:407] Connection header received
0926 16:36:02.750723 (+ 1625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:02.750747 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:02.751100 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:02.751434 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:02.752493 (+ 1059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:02.752499 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:02.752511 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:36:02.752528 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:02.752592 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:36:02.752614 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:02.752850 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:02.752956 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 16:36:02.753201 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":360,"threads_started":1}
I20260926 16:36:02.754985 26962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 236020158c58472a97d740325ee47b76: Bootstrap starting.
I20260926 16:36:02.757596 6276 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:36:02.758405 26962 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 236020158c58472a97d740325ee47b76: Neither blocks nor log segments found. Creating new log.
I20260926 16:36:02.759670 26962 log.cc:824] T 00000000000000000000000000000000 P 236020158c58472a97d740325ee47b76: Log is configured to *not* fsync() on all Append() calls
I20260926 16:36:02.762485 26962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 236020158c58472a97d740325ee47b76: No bootstrap required, opened a new log
I20260926 16:36:02.765271 26965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:02.762725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38813 (local address 127.0.0.1:45133)
0926 16:36:02.762936 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:02.762949 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:02.763349 (+ 400us) server_negotiation.cc:407] Connection header received
0926 16:36:02.763532 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:02.763536 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:02.763588 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:02.763663 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:02.764610 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:02.764614 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:02.764620 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:36:02.764634 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:02.764672 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:36:02.764688 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:02.764836 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:02.764912 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 16:36:02.764954 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 16:36:02.770706 26962 raft_consensus.cc:348] T 00000000000000000000000000000000 P 236020158c58472a97d740325ee47b76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "236020158c58472a97d740325ee47b76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45133 } }
I20260926 16:36:02.771114 26962 raft_consensus.cc:374] T 00000000000000000000000000000000 P 236020158c58472a97d740325ee47b76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:36:02.771222 26962 raft_consensus.cc:729] T 00000000000000000000000000000000 P 236020158c58472a97d740325ee47b76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 236020158c58472a97d740325ee47b76, State: Initialized, Role: FOLLOWER
I20260926 16:36:02.771857 26962 consensus_queue.cc:260] T 00000000000000000000000000000000 P 236020158c58472a97d740325ee47b76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "236020158c58472a97d740325ee47b76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45133 } }
I20260926 16:36:02.772102 26962 raft_consensus.cc:388] T 00000000000000000000000000000000 P 236020158c58472a97d740325ee47b76 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:36:02.772251 26962 raft_consensus.cc:482] T 00000000000000000000000000000000 P 236020158c58472a97d740325ee47b76 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:36:02.772426 26962 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 236020158c58472a97d740325ee47b76 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:36:02.774623 26962 raft_consensus.cc:504] T 00000000000000000000000000000000 P 236020158c58472a97d740325ee47b76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "236020158c58472a97d740325ee47b76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45133 } }
I20260926 16:36:02.775121 26962 leader_election.cc:302] T 00000000000000000000000000000000 P 236020158c58472a97d740325ee47b76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 236020158c58472a97d740325ee47b76; no voters:
I20260926 16:36:02.775733 26962 leader_election.cc:288] T 00000000000000000000000000000000 P 236020158c58472a97d740325ee47b76 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:36:02.775871 26972 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 236020158c58472a97d740325ee47b76 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:36:02.776527 26972 raft_consensus.cc:686] T 00000000000000000000000000000000 P 236020158c58472a97d740325ee47b76 [term 1 LEADER]: Becoming Leader. State: Replica: 236020158c58472a97d740325ee47b76, State: Running, Role: LEADER
I20260926 16:36:02.777104 26972 consensus_queue.cc:237] T 00000000000000000000000000000000 P 236020158c58472a97d740325ee47b76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "236020158c58472a97d740325ee47b76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45133 } }
I20260926 16:36:02.777868 26962 sys_catalog.cc:564] T 00000000000000000000000000000000 P 236020158c58472a97d740325ee47b76 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:36:02.780193 26974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 236020158c58472a97d740325ee47b76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 236020158c58472a97d740325ee47b76. Latest consensus state: current_term: 1 leader_uuid: "236020158c58472a97d740325ee47b76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "236020158c58472a97d740325ee47b76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45133 } } }
I20260926 16:36:02.780205 26973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 236020158c58472a97d740325ee47b76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "236020158c58472a97d740325ee47b76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "236020158c58472a97d740325ee47b76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45133 } } }
I20260926 16:36:02.780573 26974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 236020158c58472a97d740325ee47b76 [sys.catalog]: This master's current role is: LEADER
I20260926 16:36:02.780640 26973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 236020158c58472a97d740325ee47b76 [sys.catalog]: This master's current role is: LEADER
I20260926 16:36:02.783278 26978 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:36:02.789346 26978 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:36:02.798755 26978 catalog_manager.cc:1347] Generated new cluster ID: 898f42e3d18e43ae85c5a24aa8e0db31
I20260926 16:36:02.798883 26978 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:36:02.829919 26978 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:36:02.830422 26978 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:36:02.846202 26978 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 236020158c58472a97d740325ee47b76: Generated new TSK 0
I20260926 16:36:02.847190 26978 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:36:02.863612 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 26895
W20260926 16:36:02.864821 26957 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37449
W20260926 16:36:02.865579 6276 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 26895: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 16:36:02.865772 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 26895
2026-09-26T16:36:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1946 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440233850748-6276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:36:02 dist-test-slave-khd3 krb5kdc[26993](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:36:02 dist-test-slave-khd3 krb5kdc[26993](info): set up 2 sockets
Sep 26 16:36:02 dist-test-slave-khd3 krb5kdc[26993](info): commencing operation
krb5kdc: starting...
W20260926 16:36:03.989997 6276 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Sep 26 16:36:04 dist-test-slave-khd3 krb5kdc[26993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:36:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:36:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440233850748-6276-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440233850748-6276-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 16:36:04.259902 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440233850748-6276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440233850748-6276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440233850748-6276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45699
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440233850748-6276-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45699
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:36:04.593097 27006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:36:04.593394 27006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:36:04.593580 27006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:36:04.600922 27006 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:36:04.601066 27006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:36:04.601197 27006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:36:04.601317 27006 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:36:04.601397 27006 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:36:04.611609 27006 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440233850748-6276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440233850748-6276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45699
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440233850748-6276-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440233850748-6276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:36:04.613116 27006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:36:04.614586 27006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:36:04.626688 27011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:36:04.626816 27012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:36:04.626865 27014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:36:04.628319 27006 server_base.cc:1034] running on GCE node
I20260926 16:36:04.629251 27006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:36:04.630784 27006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:36:04.632031 27006 hybrid_clock.cc:648] HybridClock initialized: now 1790440564631980 us; error 79 us; skew 500 ppm
I20260926 16:36:04.632441 27006 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:36:04 dist-test-slave-khd3 krb5kdc[26993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:36:04.635655 27006 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 16:36:04.637375 27006 webserver.cc:460] Webserver started at http://127.0.0.1:44227/ using document root <none> and password file <none>
I20260926 16:36:04.638118 27006 fs_manager.cc:359] Metadata directory not provided
I20260926 16:36:04.638267 27006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:36:04.638588 27006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:36:04.641775 27006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440233850748-6276-0/minicluster-data/master-0/data/instance:
uuid: "c758e20433ca43ad9d29b9ddeb14b1ea"
format_stamp: "Formatted at 2026-09-26 16:36:04 on dist-test-slave-khd3"
server_key: "8ef2d9d60ccdfcac6758d8e013ea02f5"
server_key_iv: "0767c2349b8c253bca8b157474af66ef"
server_key_version: "encryptionkey@0"
I20260926 16:36:04.642617 27006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440233850748-6276-0/minicluster-data/master-0/wal/instance:
uuid: "c758e20433ca43ad9d29b9ddeb14b1ea"
format_stamp: "Formatted at 2026-09-26 16:36:04 on dist-test-slave-khd3"
server_key: "8ef2d9d60ccdfcac6758d8e013ea02f5"
server_key_iv: "0767c2349b8c253bca8b157474af66ef"
server_key_version: "encryptionkey@0"
I20260926 16:36:04.648162 27006 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260926 16:36:04.651891 27021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:36:04.653429 27006 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:36:04.653631 27006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440233850748-6276-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440233850748-6276-0/minicluster-data/master-0/wal
uuid: "c758e20433ca43ad9d29b9ddeb14b1ea"
format_stamp: "Formatted at 2026-09-26 16:36:04 on dist-test-slave-khd3"
server_key: "8ef2d9d60ccdfcac6758d8e013ea02f5"
server_key_iv: "0767c2349b8c253bca8b157474af66ef"
server_key_version: "encryptionkey@0"
I20260926 16:36:04.653851 27006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440233850748-6276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440233850748-6276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440233850748-6276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:36:04.675420 27006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:36:04.680514 27006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:36:04.680883 27006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:36:04.701730 27006 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45699
I20260926 16:36:04.701738 27072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45699 every 8 connection(s)
I20260926 16:36:04.703133 27006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
I20260926 16:36:04.704216 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 27006
I20260926 16:36:04.704399 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440233850748-6276-0/minicluster-data/master-0/wal/instance
I20260926 16:36:04.704945 6276 external_mini_cluster.cc:1428] Setting key a4d8f3fc26e7d6864d72f2ca39c028df
I20260926 16:36:04.707989 27073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:36:04.718791 27073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c758e20433ca43ad9d29b9ddeb14b1ea: Bootstrap starting.
Sep 26 16:36:04 dist-test-slave-khd3 krb5kdc[26993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 16:36:04.722470 27073 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c758e20433ca43ad9d29b9ddeb14b1ea: Neither blocks nor log segments found. Creating new log.
I20260926 16:36:04.723546 27073 log.cc:824] T 00000000000000000000000000000000 P c758e20433ca43ad9d29b9ddeb14b1ea: Log is configured to *not* fsync() on all Append() calls
I20260926 16:36:04.726533 27073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c758e20433ca43ad9d29b9ddeb14b1ea: No bootstrap required, opened a new log
I20260926 16:36:04.728082 27076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:04.706778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45683 (local address 127.0.0.1:45699)
0926 16:36:04.707543 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:04.707555 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:04.707596 (+ 41us) server_negotiation.cc:407] Connection header received
0926 16:36:04.709845 (+ 2249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:04.709870 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:04.710266 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:04.710684 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:04.712146 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:04.713604 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:04.714894 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:04.715586 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:04.715788 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:04.720325 (+ 4537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:04.720332 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:04.720347 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:36:04.720389 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:04.723808 (+ 3419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:04.724680 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:04.724693 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:04.724700 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:04.724937 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:04.725335 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:04.725340 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:04.725344 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:04.725889 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:04.726190 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:04.726652 (+ 462us) server_negotiation.cc:299] Negotiation successful
0926 16:36:04.727080 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":323,"threads_started":1}
I20260926 16:36:04.732033 6276 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:36:04.735239 27073 raft_consensus.cc:348] T 00000000000000000000000000000000 P c758e20433ca43ad9d29b9ddeb14b1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c758e20433ca43ad9d29b9ddeb14b1ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45699 } }
I20260926 16:36:04.735611 27073 raft_consensus.cc:374] T 00000000000000000000000000000000 P c758e20433ca43ad9d29b9ddeb14b1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:36:04.735719 27073 raft_consensus.cc:729] T 00000000000000000000000000000000 P c758e20433ca43ad9d29b9ddeb14b1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c758e20433ca43ad9d29b9ddeb14b1ea, State: Initialized, Role: FOLLOWER
I20260926 16:36:04.736263 27073 consensus_queue.cc:260] T 00000000000000000000000000000000 P c758e20433ca43ad9d29b9ddeb14b1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c758e20433ca43ad9d29b9ddeb14b1ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45699 } }
I20260926 16:36:04.736514 27073 raft_consensus.cc:388] T 00000000000000000000000000000000 P c758e20433ca43ad9d29b9ddeb14b1ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:36:04.736683 27073 raft_consensus.cc:482] T 00000000000000000000000000000000 P c758e20433ca43ad9d29b9ddeb14b1ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:36:04.736934 27073 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c758e20433ca43ad9d29b9ddeb14b1ea [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:36:04.739473 27073 raft_consensus.cc:504] T 00000000000000000000000000000000 P c758e20433ca43ad9d29b9ddeb14b1ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c758e20433ca43ad9d29b9ddeb14b1ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45699 } }
I20260926 16:36:04.740062 27073 leader_election.cc:302] T 00000000000000000000000000000000 P c758e20433ca43ad9d29b9ddeb14b1ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c758e20433ca43ad9d29b9ddeb14b1ea; no voters:
I20260926 16:36:04.740684 27073 leader_election.cc:288] T 00000000000000000000000000000000 P c758e20433ca43ad9d29b9ddeb14b1ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:36:04.740847 27083 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c758e20433ca43ad9d29b9ddeb14b1ea [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:36:04.741564 27083 raft_consensus.cc:686] T 00000000000000000000000000000000 P c758e20433ca43ad9d29b9ddeb14b1ea [term 1 LEADER]: Becoming Leader. State: Replica: c758e20433ca43ad9d29b9ddeb14b1ea, State: Running, Role: LEADER
I20260926 16:36:04.742146 27083 consensus_queue.cc:237] T 00000000000000000000000000000000 P c758e20433ca43ad9d29b9ddeb14b1ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c758e20433ca43ad9d29b9ddeb14b1ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45699 } }
I20260926 16:36:04.743937 27073 sys_catalog.cc:564] T 00000000000000000000000000000000 P c758e20433ca43ad9d29b9ddeb14b1ea [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:36:04.746140 27085 sys_catalog.cc:455] T 00000000000000000000000000000000 P c758e20433ca43ad9d29b9ddeb14b1ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c758e20433ca43ad9d29b9ddeb14b1ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c758e20433ca43ad9d29b9ddeb14b1ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45699 } } }
I20260926 16:36:04.746526 27085 sys_catalog.cc:458] T 00000000000000000000000000000000 P c758e20433ca43ad9d29b9ddeb14b1ea [sys.catalog]: This master's current role is: LEADER
I20260926 16:36:04.749457 27084 sys_catalog.cc:455] T 00000000000000000000000000000000 P c758e20433ca43ad9d29b9ddeb14b1ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader c758e20433ca43ad9d29b9ddeb14b1ea. Latest consensus state: current_term: 1 leader_uuid: "c758e20433ca43ad9d29b9ddeb14b1ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c758e20433ca43ad9d29b9ddeb14b1ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45699 } } }
I20260926 16:36:04.750526 27084 sys_catalog.cc:458] T 00000000000000000000000000000000 P c758e20433ca43ad9d29b9ddeb14b1ea [sys.catalog]: This master's current role is: LEADER
I20260926 16:36:04.753145 27088 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:36:04.755308 27076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:04.736657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55581 (local address 127.0.0.1:45699)
0926 16:36:04.736871 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:04.736881 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:04.737391 (+ 510us) server_negotiation.cc:407] Connection header received
0926 16:36:04.737514 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:04.737525 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:04.737608 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:04.737712 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:04.739108 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:04.740192 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:04.742161 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:04.742844 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:04.742967 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:04.746483 (+ 3516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:04.746489 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:04.746494 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:36:04.746543 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:04.751390 (+ 4847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:04.752500 (+ 1110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:04.752511 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:04.752514 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:04.752794 (+ 280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:04.753360 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:04.753367 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:04.753372 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:04.753844 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:04.754183 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:04.754454 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 16:36:04.754694 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 16:36:04.760401 27088 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:36:04.770870 27088 catalog_manager.cc:1347] Generated new cluster ID: ea5474f722d5450690dc6e2a7df32eee
I20260926 16:36:04.771044 27088 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:36:04.782218 27088 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:36:04.784231 27088 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:36:04.795838 27088 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c758e20433ca43ad9d29b9ddeb14b1ea: Generated new TSK 0
I20260926 16:36:04.796821 27088 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:36:04.807582 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 27006
I20260926 16:36:04.808732 27068 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:36:04.899542 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 27006
2026-09-26T16:36:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2033 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440233850748-6276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:36:04 dist-test-slave-khd3 krb5kdc[27105](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:36:04 dist-test-slave-khd3 krb5kdc[27105](info): set up 2 sockets
Sep 26 16:36:04 dist-test-slave-khd3 krb5kdc[27105](info): commencing operation
krb5kdc: starting...
W20260926 16:36:06.010335 6276 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Sep 26 16:36:06 dist-test-slave-khd3 krb5kdc[27105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440566, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:36:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:36:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440233850748-6276-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440233850748-6276-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 16:36:06.283919 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440233850748-6276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440233850748-6276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440233850748-6276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37907
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440233850748-6276-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37907
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:36:06.624666 27118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:36:06.625008 27118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:36:06.625223 27118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:36:06.632664 27118 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:36:06.632812 27118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:36:06.632903 27118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:36:06.633009 27118 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:36:06.633093 27118 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:36:06.643335 27118 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440233850748-6276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440233850748-6276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37907
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440233850748-6276-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440233850748-6276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:36:06.644681 27118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:36:06.646049 27118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:36:06.658360 27123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:36:06.658409 27126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:36:06.658326 27124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:36:06.659130 27118 server_base.cc:1034] running on GCE node
I20260926 16:36:06.659991 27118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:36:06.661538 27118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:36:06.662765 27118 hybrid_clock.cc:648] HybridClock initialized: now 1790440566662696 us; error 60 us; skew 500 ppm
I20260926 16:36:06.663162 27118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:36:06 dist-test-slave-khd3 krb5kdc[27105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440566, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:36:06.666477 27118 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 16:36:06.668182 27118 webserver.cc:460] Webserver started at http://127.0.0.1:36099/ using document root <none> and password file <none>
I20260926 16:36:06.668990 27118 fs_manager.cc:359] Metadata directory not provided
I20260926 16:36:06.669154 27118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:36:06.669467 27118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:36:06.672500 27118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440233850748-6276-0/minicluster-data/master-0/data/instance:
uuid: "f12b59512eba4414a0742f4f16917bab"
format_stamp: "Formatted at 2026-09-26 16:36:06 on dist-test-slave-khd3"
server_key: "f22ac2734ea26842dbf25f5ebfcf2c20"
server_key_iv: "154ce5f630830cf45574e0d709f687f5"
server_key_version: "encryptionkey@0"
I20260926 16:36:06.673355 27118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440233850748-6276-0/minicluster-data/master-0/wal/instance:
uuid: "f12b59512eba4414a0742f4f16917bab"
format_stamp: "Formatted at 2026-09-26 16:36:06 on dist-test-slave-khd3"
server_key: "f22ac2734ea26842dbf25f5ebfcf2c20"
server_key_iv: "154ce5f630830cf45574e0d709f687f5"
server_key_version: "encryptionkey@0"
I20260926 16:36:06.678870 27118 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 16:36:06.682713 27133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:36:06.684098 27118 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 16:36:06.684300 27118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440233850748-6276-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440233850748-6276-0/minicluster-data/master-0/wal
uuid: "f12b59512eba4414a0742f4f16917bab"
format_stamp: "Formatted at 2026-09-26 16:36:06 on dist-test-slave-khd3"
server_key: "f22ac2734ea26842dbf25f5ebfcf2c20"
server_key_iv: "154ce5f630830cf45574e0d709f687f5"
server_key_version: "encryptionkey@0"
I20260926 16:36:06.684514 27118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440233850748-6276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440233850748-6276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440233850748-6276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:36:06.718525 27118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:36:06.723050 27118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:36:06.723383 27118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:36:06.742887 27118 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37907
I20260926 16:36:06.742918 27184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37907 every 8 connection(s)
I20260926 16:36:06.744321 27118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
I20260926 16:36:06.748041 27185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:36:06.749969 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 27118
I20260926 16:36:06.750162 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440233850748-6276-0/minicluster-data/master-0/wal/instance
I20260926 16:36:06.750701 6276 external_mini_cluster.cc:1428] Setting key d800e85964884268f1d8757495e5060a
I20260926 16:36:06.761299 27185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f12b59512eba4414a0742f4f16917bab: Bootstrap starting.
I20260926 16:36:06.763489 27188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:06.752597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50549 (local address 127.0.0.1:37907)
0926 16:36:06.753414 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:06.753429 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:06.753471 (+ 42us) server_negotiation.cc:407] Connection header received
0926 16:36:06.755013 (+ 1542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:06.755038 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:06.755453 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:06.755849 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:06.756386 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:06.757717 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:06.759041 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:06.759757 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:06.759951 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:06.761196 (+ 1245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:06.761203 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:06.761218 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:36:06.761815 (+ 597us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:06.761902 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:36:06.761927 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:06.762051 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:06.762278 (+ 227us) server_negotiation.cc:299] Negotiation successful
0926 16:36:06.762529 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":353,"threads_started":1}
I20260926 16:36:06.764611 27185 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f12b59512eba4414a0742f4f16917bab: Neither blocks nor log segments found. Creating new log.
I20260926 16:36:06.765717 27185 log.cc:824] T 00000000000000000000000000000000 P f12b59512eba4414a0742f4f16917bab: Log is configured to *not* fsync() on all Append() calls
I20260926 16:36:06.767170 6276 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:36:06.769021 27185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f12b59512eba4414a0742f4f16917bab: No bootstrap required, opened a new log
I20260926 16:36:06.777853 27188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:06.771476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41529 (local address 127.0.0.1:37907)
0926 16:36:06.771647 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:06.771651 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:06.772139 (+ 488us) server_negotiation.cc:407] Connection header received
0926 16:36:06.772351 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:06.772355 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:06.772413 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:06.772527 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:06.773080 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:06.773822 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:06.775222 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:06.775696 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:06.775862 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:06.777057 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:06.777062 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:06.777065 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:36:06.777085 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:06.777127 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:36:06.777143 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:06.777241 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:06.777381 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 16:36:06.777473 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 16:36:06.777629 27185 raft_consensus.cc:348] T 00000000000000000000000000000000 P f12b59512eba4414a0742f4f16917bab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f12b59512eba4414a0742f4f16917bab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37907 } }
I20260926 16:36:06.778211 27185 raft_consensus.cc:374] T 00000000000000000000000000000000 P f12b59512eba4414a0742f4f16917bab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:36:06.778342 27185 raft_consensus.cc:729] T 00000000000000000000000000000000 P f12b59512eba4414a0742f4f16917bab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12b59512eba4414a0742f4f16917bab, State: Initialized, Role: FOLLOWER
I20260926 16:36:06.778957 27185 consensus_queue.cc:260] T 00000000000000000000000000000000 P f12b59512eba4414a0742f4f16917bab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f12b59512eba4414a0742f4f16917bab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37907 } }
I20260926 16:36:06.779206 27185 raft_consensus.cc:388] T 00000000000000000000000000000000 P f12b59512eba4414a0742f4f16917bab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:36:06.779378 27185 raft_consensus.cc:482] T 00000000000000000000000000000000 P f12b59512eba4414a0742f4f16917bab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:36:06.779567 27185 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f12b59512eba4414a0742f4f16917bab [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:36:06.781844 27185 raft_consensus.cc:504] T 00000000000000000000000000000000 P f12b59512eba4414a0742f4f16917bab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f12b59512eba4414a0742f4f16917bab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37907 } }
I20260926 16:36:06.782339 27185 leader_election.cc:302] T 00000000000000000000000000000000 P f12b59512eba4414a0742f4f16917bab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f12b59512eba4414a0742f4f16917bab; no voters:
I20260926 16:36:06.782881 27185 leader_election.cc:288] T 00000000000000000000000000000000 P f12b59512eba4414a0742f4f16917bab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:36:06.783028 27195 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f12b59512eba4414a0742f4f16917bab [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:36:06.783782 27195 raft_consensus.cc:686] T 00000000000000000000000000000000 P f12b59512eba4414a0742f4f16917bab [term 1 LEADER]: Becoming Leader. State: Replica: f12b59512eba4414a0742f4f16917bab, State: Running, Role: LEADER
I20260926 16:36:06.784704 27195 consensus_queue.cc:237] T 00000000000000000000000000000000 P f12b59512eba4414a0742f4f16917bab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f12b59512eba4414a0742f4f16917bab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37907 } }
I20260926 16:36:06.785316 27185 sys_catalog.cc:564] T 00000000000000000000000000000000 P f12b59512eba4414a0742f4f16917bab [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:36:06.787477 27196 sys_catalog.cc:455] T 00000000000000000000000000000000 P f12b59512eba4414a0742f4f16917bab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f12b59512eba4414a0742f4f16917bab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f12b59512eba4414a0742f4f16917bab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37907 } } }
I20260926 16:36:06.787879 27196 sys_catalog.cc:458] T 00000000000000000000000000000000 P f12b59512eba4414a0742f4f16917bab [sys.catalog]: This master's current role is: LEADER
I20260926 16:36:06.787719 27197 sys_catalog.cc:455] T 00000000000000000000000000000000 P f12b59512eba4414a0742f4f16917bab [sys.catalog]: SysCatalogTable state changed. Reason: New leader f12b59512eba4414a0742f4f16917bab. Latest consensus state: current_term: 1 leader_uuid: "f12b59512eba4414a0742f4f16917bab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f12b59512eba4414a0742f4f16917bab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37907 } } }
I20260926 16:36:06.788110 27197 sys_catalog.cc:458] T 00000000000000000000000000000000 P f12b59512eba4414a0742f4f16917bab [sys.catalog]: This master's current role is: LEADER
I20260926 16:36:06.790755 27203 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:36:06.795802 27203 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:36:06.805300 27203 catalog_manager.cc:1347] Generated new cluster ID: bd9a2ff99fe0485cb0d6cb425825fd3b
I20260926 16:36:06.805430 27203 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:36:06.818256 27203 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:36:06.819679 27203 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:36:06.827880 27203 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f12b59512eba4414a0742f4f16917bab: Generated new TSK 0
I20260926 16:36:06.829066 27203 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:36:06.840919 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 27118
W20260926 16:36:06.842540 27180 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:50549
W20260926 16:36:06.843374 6276 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 27118: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 16:36:06.843526 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 27118
2026-09-26T16:36:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1941 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440233850748-6276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:36:06 dist-test-slave-khd3 krb5kdc[27216](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:36:06 dist-test-slave-khd3 krb5kdc[27216](info): set up 2 sockets
Sep 26 16:36:06 dist-test-slave-khd3 krb5kdc[27216](info): commencing operation
krb5kdc: starting...
W20260926 16:36:07.954206 6276 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Sep 26 16:36:08 dist-test-slave-khd3 krb5kdc[27216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440568, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:36:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:36:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440233850748-6276-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440233850748-6276-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 16:36:08.221048 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440233850748-6276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440233850748-6276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440233850748-6276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34725
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440233850748-6276-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:36:08.562438 27229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:36:08.562764 27229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:36:08.562860 27229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:36:08.570498 27229 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:36:08.570639 27229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:36:08.570688 27229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:36:08.570742 27229 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:36:08.570806 27229 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:36:08.581049 27229 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440233850748-6276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440233850748-6276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440233850748-6276-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440233850748-6276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:36:08.582368 27229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:36:08.583796 27229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:36:08.596549 27237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:36:08.596623 27234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:36:08.596586 27235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:36:08.597519 27229 server_base.cc:1034] running on GCE node
I20260926 16:36:08.598414 27229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:36:08.599990 27229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:36:08.601233 27229 hybrid_clock.cc:648] HybridClock initialized: now 1790440568601187 us; error 46 us; skew 500 ppm
I20260926 16:36:08.601606 27229 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:36:08 dist-test-slave-khd3 krb5kdc[27216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440568, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:36:08.618520 27229 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 16:36:08.620357 27229 webserver.cc:460] Webserver started at http://127.0.0.1:41139/ using document root <none> and password file <none>
I20260926 16:36:08.621254 27229 fs_manager.cc:359] Metadata directory not provided
I20260926 16:36:08.621413 27229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:36:08.621793 27229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:36:08.625108 27229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440233850748-6276-0/minicluster-data/master-0/data/instance:
uuid: "c9779638e6074088a5bf970786744174"
format_stamp: "Formatted at 2026-09-26 16:36:08 on dist-test-slave-khd3"
server_key: "6e3fe7f72418a85b6072959b95be6b6c"
server_key_iv: "b82638e1ec4fd356b8374e9d4779e313"
server_key_version: "encryptionkey@0"
I20260926 16:36:08.625909 27229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440233850748-6276-0/minicluster-data/master-0/wal/instance:
uuid: "c9779638e6074088a5bf970786744174"
format_stamp: "Formatted at 2026-09-26 16:36:08 on dist-test-slave-khd3"
server_key: "6e3fe7f72418a85b6072959b95be6b6c"
server_key_iv: "b82638e1ec4fd356b8374e9d4779e313"
server_key_version: "encryptionkey@0"
I20260926 16:36:08.631803 27229 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 16:36:08.635835 27244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:36:08.637563 27229 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:36:08.637795 27229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440233850748-6276-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440233850748-6276-0/minicluster-data/master-0/wal
uuid: "c9779638e6074088a5bf970786744174"
format_stamp: "Formatted at 2026-09-26 16:36:08 on dist-test-slave-khd3"
server_key: "6e3fe7f72418a85b6072959b95be6b6c"
server_key_iv: "b82638e1ec4fd356b8374e9d4779e313"
server_key_version: "encryptionkey@0"
I20260926 16:36:08.638028 27229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440233850748-6276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440233850748-6276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440233850748-6276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:36:08.644300 27229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:36:08.648658 27229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:36:08.649008 27229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:36:08.669011 27229 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34725
I20260926 16:36:08.669050 27295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34725 every 8 connection(s)
I20260926 16:36:08.670486 27229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
I20260926 16:36:08.674476 27296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:36:08.676998 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 27229
I20260926 16:36:08.677217 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440233850748-6276-0/minicluster-data/master-0/wal/instance
I20260926 16:36:08.677670 6276 external_mini_cluster.cc:1428] Setting key 4415cddd0e3282714a58bfb1bf944146
I20260926 16:36:08.687121 27299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:08.679625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40757 (local address 127.0.0.1:34725)
0926 16:36:08.680483 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:08.680499 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:08.680556 (+ 57us) server_negotiation.cc:407] Connection header received
0926 16:36:08.682940 (+ 2384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:08.682973 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:08.683451 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:08.683942 (+ 491us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:08.685003 (+ 1061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:08.685013 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:08.685036 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:36:08.685053 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:08.685146 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:36:08.685175 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:08.685408 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:08.685547 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 16:36:08.685771 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":348,"threads_started":1}
I20260926 16:36:08.689069 27296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9779638e6074088a5bf970786744174: Bootstrap starting.
I20260926 16:36:08.691111 6276 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:36:08.693231 27296 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9779638e6074088a5bf970786744174: Neither blocks nor log segments found. Creating new log.
I20260926 16:36:08.694715 27296 log.cc:824] T 00000000000000000000000000000000 P c9779638e6074088a5bf970786744174: Log is configured to *not* fsync() on all Append() calls
I20260926 16:36:08.698539 27296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9779638e6074088a5bf970786744174: No bootstrap required, opened a new log
I20260926 16:36:08.699313 27299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:08.696534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60795 (local address 127.0.0.1:34725)
0926 16:36:08.696750 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:08.696779 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:08.697103 (+ 324us) server_negotiation.cc:407] Connection header received
0926 16:36:08.697288 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:08.697292 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:08.697342 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:08.697424 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:08.698494 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:08.698498 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:08.698501 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:36:08.698516 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:08.698559 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:36:08.698575 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:08.698753 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:08.698892 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 16:36:08.698958 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 16:36:08.706374 27296 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9779638e6074088a5bf970786744174 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9779638e6074088a5bf970786744174" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34725 } }
I20260926 16:36:08.706823 27296 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9779638e6074088a5bf970786744174 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:36:08.706936 27296 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9779638e6074088a5bf970786744174 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9779638e6074088a5bf970786744174, State: Initialized, Role: FOLLOWER
I20260926 16:36:08.707619 27296 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9779638e6074088a5bf970786744174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9779638e6074088a5bf970786744174" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34725 } }
I20260926 16:36:08.707880 27296 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9779638e6074088a5bf970786744174 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:36:08.708026 27296 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9779638e6074088a5bf970786744174 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:36:08.708204 27296 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9779638e6074088a5bf970786744174 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:36:08.710532 27296 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9779638e6074088a5bf970786744174 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9779638e6074088a5bf970786744174" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34725 } }
I20260926 16:36:08.711058 27296 leader_election.cc:302] T 00000000000000000000000000000000 P c9779638e6074088a5bf970786744174 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9779638e6074088a5bf970786744174; no voters:
I20260926 16:36:08.711683 27296 leader_election.cc:288] T 00000000000000000000000000000000 P c9779638e6074088a5bf970786744174 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:36:08.711824 27306 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9779638e6074088a5bf970786744174 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:36:08.712566 27306 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9779638e6074088a5bf970786744174 [term 1 LEADER]: Becoming Leader. State: Replica: c9779638e6074088a5bf970786744174, State: Running, Role: LEADER
I20260926 16:36:08.713212 27306 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9779638e6074088a5bf970786744174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9779638e6074088a5bf970786744174" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34725 } }
I20260926 16:36:08.713903 27296 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9779638e6074088a5bf970786744174 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:36:08.716384 27307 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9779638e6074088a5bf970786744174 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9779638e6074088a5bf970786744174" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9779638e6074088a5bf970786744174" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34725 } } }
I20260926 16:36:08.716821 27307 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9779638e6074088a5bf970786744174 [sys.catalog]: This master's current role is: LEADER
I20260926 16:36:08.716502 27308 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9779638e6074088a5bf970786744174 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9779638e6074088a5bf970786744174. Latest consensus state: current_term: 1 leader_uuid: "c9779638e6074088a5bf970786744174" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9779638e6074088a5bf970786744174" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34725 } } }
I20260926 16:36:08.716957 27308 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9779638e6074088a5bf970786744174 [sys.catalog]: This master's current role is: LEADER
I20260926 16:36:08.718581 27311 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:36:08.724368 27311 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:36:08.733856 27311 catalog_manager.cc:1347] Generated new cluster ID: 2d35909ad2064b74922c9d93cec5462e
I20260926 16:36:08.734026 27311 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:36:08.776770 27311 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:36:08.777369 27311 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:36:08.784792 27311 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c9779638e6074088a5bf970786744174: Generated new TSK 0
I20260926 16:36:08.785725 27311 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:36:08.796972 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 27229
W20260926 16:36:08.798197 27291 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40757
W20260926 16:36:08.798898 6276 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 27229: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 16:36:08.799057 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 27229
2026-09-26T16:36:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1958 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440233850748-6276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:36:08 dist-test-slave-khd3 krb5kdc[27327](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:36:08 dist-test-slave-khd3 krb5kdc[27327](info): set up 2 sockets
Sep 26 16:36:08 dist-test-slave-khd3 krb5kdc[27327](info): commencing operation
krb5kdc: starting...
W20260926 16:36:09.906111 6276 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Sep 26 16:36:10 dist-test-slave-khd3 krb5kdc[27327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:36:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:36:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440233850748-6276-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440233850748-6276-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 16:36:10.172704 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440233850748-6276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440233850748-6276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440233850748-6276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40793
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440233850748-6276-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40793
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:36:10.511875 27340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:36:10.512194 27340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:36:10.512356 27340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:36:10.519831 27340 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:36:10.519951 27340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:36:10.520071 27340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:36:10.520174 27340 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:36:10.520256 27340 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:36:10.530344 27340 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440233850748-6276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440233850748-6276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40793
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440233850748-6276-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440233850748-6276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:36:10.531631 27340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:36:10.533061 27340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:36:10.544965 27346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:36:10.544919 27348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:36:10.545007 27345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:36:10.545907 27340 server_base.cc:1034] running on GCE node
I20260926 16:36:10.546742 27340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:36:10.548313 27340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:36:10.549539 27340 hybrid_clock.cc:648] HybridClock initialized: now 1790440570549486 us; error 51 us; skew 500 ppm
I20260926 16:36:10.549901 27340 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:36:10 dist-test-slave-khd3 krb5kdc[27327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440570, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:36:10.552968 27340 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 16:36:10.554445 27340 webserver.cc:460] Webserver started at http://127.0.0.1:45581/ using document root <none> and password file <none>
I20260926 16:36:10.555157 27340 fs_manager.cc:359] Metadata directory not provided
I20260926 16:36:10.555300 27340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:36:10.555603 27340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:36:10.558772 27340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440233850748-6276-0/minicluster-data/master-0/data/instance:
uuid: "6a13d173e67a458387bf58863c2e7312"
format_stamp: "Formatted at 2026-09-26 16:36:10 on dist-test-slave-khd3"
server_key: "8e932b1f0a8cbb8e1fc08a813d77b4d8"
server_key_iv: "3a0d46e8e05c34a4db41db96cd83c28b"
server_key_version: "encryptionkey@0"
I20260926 16:36:10.559620 27340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440233850748-6276-0/minicluster-data/master-0/wal/instance:
uuid: "6a13d173e67a458387bf58863c2e7312"
format_stamp: "Formatted at 2026-09-26 16:36:10 on dist-test-slave-khd3"
server_key: "8e932b1f0a8cbb8e1fc08a813d77b4d8"
server_key_iv: "3a0d46e8e05c34a4db41db96cd83c28b"
server_key_version: "encryptionkey@0"
I20260926 16:36:10.565024 27340 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260926 16:36:10.568666 27355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:36:10.570212 27340 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:36:10.570415 27340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440233850748-6276-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440233850748-6276-0/minicluster-data/master-0/wal
uuid: "6a13d173e67a458387bf58863c2e7312"
format_stamp: "Formatted at 2026-09-26 16:36:10 on dist-test-slave-khd3"
server_key: "8e932b1f0a8cbb8e1fc08a813d77b4d8"
server_key_iv: "3a0d46e8e05c34a4db41db96cd83c28b"
server_key_version: "encryptionkey@0"
I20260926 16:36:10.570626 27340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440233850748-6276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440233850748-6276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440233850748-6276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:36:10.589941 27340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:36:10.594446 27340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:36:10.594787 27340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:36:10.614841 27340 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40793
I20260926 16:36:10.614864 27406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40793 every 8 connection(s)
I20260926 16:36:10.616395 27340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
I20260926 16:36:10.618917 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 27340
I20260926 16:36:10.619110 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440233850748-6276-0/minicluster-data/master-0/wal/instance
I20260926 16:36:10.619560 6276 external_mini_cluster.cc:1428] Setting key a4b9013520a691a435eaa0ab175d9ef2
I20260926 16:36:10.620934 27407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 16:36:10 dist-test-slave-khd3 krb5kdc[27327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 16:36:10.635002 27407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a13d173e67a458387bf58863c2e7312: Bootstrap starting.
I20260926 16:36:10.638445 27407 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a13d173e67a458387bf58863c2e7312: Neither blocks nor log segments found. Creating new log.
I20260926 16:36:10.639722 27407 log.cc:824] T 00000000000000000000000000000000 P 6a13d173e67a458387bf58863c2e7312: Log is configured to *not* fsync() on all Append() calls
I20260926 16:36:10.643088 27407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a13d173e67a458387bf58863c2e7312: No bootstrap required, opened a new log
I20260926 16:36:10.643468 27410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:10.621407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.235:52907 (local address 127.0.0.1:40793)
0926 16:36:10.622228 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:10.622244 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:10.622295 (+ 51us) server_negotiation.cc:407] Connection header received
0926 16:36:10.624312 (+ 2017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:10.624341 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:10.624926 (+ 585us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:10.625405 (+ 479us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:10.626660 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:10.628722 (+ 2062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:10.629967 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:10.631011 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:10.631224 (+ 213us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:10.635625 (+ 4401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:10.635639 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:10.635654 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:36:10.635691 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:10.638910 (+ 3219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:10.639887 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:10.639897 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:10.639911 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:10.640081 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:10.640673 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:10.640678 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:10.640683 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:10.641359 (+ 676us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:10.641588 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:10.642094 (+ 506us) server_negotiation.cc:299] Negotiation successful
0926 16:36:10.642390 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":341,"threads_started":1}
I20260926 16:36:10.648008 6276 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:36:10.652067 27407 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a13d173e67a458387bf58863c2e7312 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a13d173e67a458387bf58863c2e7312" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40793 } }
I20260926 16:36:10.652681 27407 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a13d173e67a458387bf58863c2e7312 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:36:10.652855 27407 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a13d173e67a458387bf58863c2e7312 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a13d173e67a458387bf58863c2e7312, State: Initialized, Role: FOLLOWER
I20260926 16:36:10.653563 27407 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a13d173e67a458387bf58863c2e7312 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a13d173e67a458387bf58863c2e7312" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40793 } }
I20260926 16:36:10.653826 27407 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a13d173e67a458387bf58863c2e7312 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:36:10.653991 27407 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a13d173e67a458387bf58863c2e7312 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:36:10.654188 27407 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a13d173e67a458387bf58863c2e7312 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:36:10.657284 27407 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a13d173e67a458387bf58863c2e7312 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a13d173e67a458387bf58863c2e7312" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40793 } }
I20260926 16:36:10.658008 27407 leader_election.cc:302] T 00000000000000000000000000000000 P 6a13d173e67a458387bf58863c2e7312 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a13d173e67a458387bf58863c2e7312; no voters:
I20260926 16:36:10.658798 27407 leader_election.cc:288] T 00000000000000000000000000000000 P 6a13d173e67a458387bf58863c2e7312 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:36:10.659001 27417 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a13d173e67a458387bf58863c2e7312 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:36:10.661399 27407 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a13d173e67a458387bf58863c2e7312 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:36:10.664881 27417 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a13d173e67a458387bf58863c2e7312 [term 1 LEADER]: Becoming Leader. State: Replica: 6a13d173e67a458387bf58863c2e7312, State: Running, Role: LEADER
I20260926 16:36:10.665779 27417 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a13d173e67a458387bf58863c2e7312 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a13d173e67a458387bf58863c2e7312" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40793 } }
I20260926 16:36:10.670073 27418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a13d173e67a458387bf58863c2e7312 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a13d173e67a458387bf58863c2e7312" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a13d173e67a458387bf58863c2e7312" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40793 } } }
I20260926 16:36:10.670643 27418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a13d173e67a458387bf58863c2e7312 [sys.catalog]: This master's current role is: LEADER
I20260926 16:36:10.671326 27419 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a13d173e67a458387bf58863c2e7312 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a13d173e67a458387bf58863c2e7312. Latest consensus state: current_term: 1 leader_uuid: "6a13d173e67a458387bf58863c2e7312" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a13d173e67a458387bf58863c2e7312" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40793 } } }
I20260926 16:36:10.671722 27419 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a13d173e67a458387bf58863c2e7312 [sys.catalog]: This master's current role is: LEADER
I20260926 16:36:10.674193 27410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:10.652643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.235:47631 (local address 127.0.0.1:40793)
0926 16:36:10.652910 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:10.652919 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:10.653511 (+ 592us) server_negotiation.cc:407] Connection header received
0926 16:36:10.653730 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:10.653735 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:10.653850 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:10.653971 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:10.655841 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:10.656679 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:10.658643 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:10.659739 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:10.660883 (+ 1144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:10.664552 (+ 3669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:10.664559 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:10.664564 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:36:10.664613 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:10.668964 (+ 4351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:10.670282 (+ 1318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:10.670288 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:10.670295 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:10.670551 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:10.671244 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:10.671250 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:10.671255 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:10.671802 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:10.673143 (+ 1341us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:10.673482 (+ 339us) server_negotiation.cc:299] Negotiation successful
0926 16:36:10.673732 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260926 16:36:10.678737 27431 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6a13d173e67a458387bf58863c2e7312: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 16:36:10.678864 27431 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 16:36:10.679062 27432 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:36:10.681306 27432 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:36:10.690866 27432 catalog_manager.cc:1347] Generated new cluster ID: be6dd4bc089046efba7f68785e721e64
I20260926 16:36:10.691007 27432 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:36:10.701843 27432 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:36:10.703550 27432 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:36:10.712718 27432 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a13d173e67a458387bf58863c2e7312: Generated new TSK 0
I20260926 16:36:10.713681 27432 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:36:10.725728 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 27340
I20260926 16:36:10.726842 27402 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:36:10.820226 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 27340
W20260926 16:36:10.838318 27325 connection.cc:535] client connection to 127.0.0.1:40793 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40793): Connection reset by peer (error 104)
2026-09-26T16:36:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2021 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440233850748-6276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:36:10 dist-test-slave-khd3 krb5kdc[27439](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:36:10 dist-test-slave-khd3 krb5kdc[27439](info): set up 2 sockets
Sep 26 16:36:10 dist-test-slave-khd3 krb5kdc[27439](info): commencing operation
krb5kdc: starting...
W20260926 16:36:11.938143 6276 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Sep 26 16:36:12 dist-test-slave-khd3 krb5kdc[27439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:36:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:36:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440233850748-6276-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440233850748-6276-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 16:36:12.216194 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440233850748-6276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440233850748-6276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440233850748-6276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46257
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440233850748-6276-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46257
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:36:12.551009 27452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:36:12.551311 27452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:36:12.551405 27452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:36:12.558847 27452 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:36:12.558974 27452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:36:12.559021 27452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:36:12.559089 27452 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:36:12.559134 27452 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:36:12.569434 27452 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440233850748-6276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440233850748-6276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46257
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440233850748-6276-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440233850748-6276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:36:12.570832 27452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:36:12.572130 27452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:36:12.584242 27458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:36:12.584393 27460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:36:12.584353 27457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:36:12.585423 27452 server_base.cc:1034] running on GCE node
I20260926 16:36:12.586292 27452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:36:12.587806 27452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:36:12.589044 27452 hybrid_clock.cc:648] HybridClock initialized: now 1790440572588971 us; error 75 us; skew 500 ppm
I20260926 16:36:12.589427 27452 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:36:12 dist-test-slave-khd3 krb5kdc[27439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:36:12.592662 27452 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 16:36:12.594180 27452 webserver.cc:460] Webserver started at http://127.0.0.1:35221/ using document root <none> and password file <none>
I20260926 16:36:12.594825 27452 fs_manager.cc:359] Metadata directory not provided
I20260926 16:36:12.594925 27452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:36:12.595185 27452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:36:12.598646 27452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440233850748-6276-0/minicluster-data/master-0/data/instance:
uuid: "2512db3e6009416dbbcd437baf39d0f6"
format_stamp: "Formatted at 2026-09-26 16:36:12 on dist-test-slave-khd3"
server_key: "7e985b4b7610fe67ffab7462a8181d43"
server_key_iv: "dbce590b6e3f3baf6ad205d34f38b002"
server_key_version: "encryptionkey@0"
I20260926 16:36:12.599368 27452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440233850748-6276-0/minicluster-data/master-0/wal/instance:
uuid: "2512db3e6009416dbbcd437baf39d0f6"
format_stamp: "Formatted at 2026-09-26 16:36:12 on dist-test-slave-khd3"
server_key: "7e985b4b7610fe67ffab7462a8181d43"
server_key_iv: "dbce590b6e3f3baf6ad205d34f38b002"
server_key_version: "encryptionkey@0"
I20260926 16:36:12.604529 27452 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 16:36:12.608167 27467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:36:12.609723 27452 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:36:12.609905 27452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440233850748-6276-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440233850748-6276-0/minicluster-data/master-0/wal
uuid: "2512db3e6009416dbbcd437baf39d0f6"
format_stamp: "Formatted at 2026-09-26 16:36:12 on dist-test-slave-khd3"
server_key: "7e985b4b7610fe67ffab7462a8181d43"
server_key_iv: "dbce590b6e3f3baf6ad205d34f38b002"
server_key_version: "encryptionkey@0"
I20260926 16:36:12.610136 27452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440233850748-6276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440233850748-6276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440233850748-6276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:36:12.632100 27452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:36:12.636940 27452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:36:12.637333 27452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:36:12.657068 27452 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46257
I20260926 16:36:12.657171 27518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46257 every 8 connection(s)
I20260926 16:36:12.658545 27452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
I20260926 16:36:12.661736 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 27452
I20260926 16:36:12.661922 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440233850748-6276-0/minicluster-data/master-0/wal/instance
I20260926 16:36:12.662458 6276 external_mini_cluster.cc:1428] Setting key 54b271615c3ad44dd5815e4882323769
I20260926 16:36:12.662688 27519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:36:12.677520 27522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:12.664326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.235:49347 (local address 127.0.0.1:46257)
0926 16:36:12.665096 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:12.665107 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:12.665158 (+ 51us) server_negotiation.cc:407] Connection header received
0926 16:36:12.667297 (+ 2139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:12.667319 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:12.667732 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:12.668074 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:12.668752 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:12.670499 (+ 1747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:12.672406 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:12.673223 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:12.673377 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:12.674862 (+ 1485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:12.674872 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:12.674888 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:36:12.675529 (+ 641us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:12.675615 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:36:12.675641 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:12.675767 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:12.676167 (+ 400us) server_negotiation.cc:299] Negotiation successful
0926 16:36:12.676373 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":365,"threads_started":1}
I20260926 16:36:12.677634 27519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2512db3e6009416dbbcd437baf39d0f6: Bootstrap starting.
I20260926 16:36:12.681047 27519 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2512db3e6009416dbbcd437baf39d0f6: Neither blocks nor log segments found. Creating new log.
I20260926 16:36:12.681682 6276 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:36:12.682324 27519 log.cc:824] T 00000000000000000000000000000000 P 2512db3e6009416dbbcd437baf39d0f6: Log is configured to *not* fsync() on all Append() calls
I20260926 16:36:12.685956 27519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2512db3e6009416dbbcd437baf39d0f6: No bootstrap required, opened a new log
I20260926 16:36:12.693817 27522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:12.686782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.235:35023 (local address 127.0.0.1:46257)
0926 16:36:12.686975 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:12.686981 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:12.687367 (+ 386us) server_negotiation.cc:407] Connection header received
0926 16:36:12.687536 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:12.687542 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:12.687631 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:12.687745 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:12.688258 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:12.689486 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:12.690895 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:12.691648 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:12.691785 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:12.692971 (+ 1186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:12.692977 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:12.692983 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:36:12.693006 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:12.693050 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:36:12.693069 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:12.693180 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:12.693396 (+ 216us) server_negotiation.cc:299] Negotiation successful
0926 16:36:12.693449 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 16:36:12.695581 27519 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2512db3e6009416dbbcd437baf39d0f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2512db3e6009416dbbcd437baf39d0f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46257 } }
I20260926 16:36:12.695984 27519 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2512db3e6009416dbbcd437baf39d0f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:36:12.696112 27519 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2512db3e6009416dbbcd437baf39d0f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2512db3e6009416dbbcd437baf39d0f6, State: Initialized, Role: FOLLOWER
I20260926 16:36:12.696832 27519 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2512db3e6009416dbbcd437baf39d0f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2512db3e6009416dbbcd437baf39d0f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46257 } }
I20260926 16:36:12.697165 27519 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2512db3e6009416dbbcd437baf39d0f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:36:12.697348 27519 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2512db3e6009416dbbcd437baf39d0f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:36:12.697566 27519 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2512db3e6009416dbbcd437baf39d0f6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:36:12.699723 27519 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2512db3e6009416dbbcd437baf39d0f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2512db3e6009416dbbcd437baf39d0f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46257 } }
I20260926 16:36:12.700237 27519 leader_election.cc:302] T 00000000000000000000000000000000 P 2512db3e6009416dbbcd437baf39d0f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2512db3e6009416dbbcd437baf39d0f6; no voters:
I20260926 16:36:12.700915 27519 leader_election.cc:288] T 00000000000000000000000000000000 P 2512db3e6009416dbbcd437baf39d0f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:36:12.701045 27529 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2512db3e6009416dbbcd437baf39d0f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:36:12.701776 27529 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2512db3e6009416dbbcd437baf39d0f6 [term 1 LEADER]: Becoming Leader. State: Replica: 2512db3e6009416dbbcd437baf39d0f6, State: Running, Role: LEADER
I20260926 16:36:12.702627 27529 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2512db3e6009416dbbcd437baf39d0f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2512db3e6009416dbbcd437baf39d0f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46257 } }
I20260926 16:36:12.703255 27519 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2512db3e6009416dbbcd437baf39d0f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:36:12.706252 27531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2512db3e6009416dbbcd437baf39d0f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2512db3e6009416dbbcd437baf39d0f6. Latest consensus state: current_term: 1 leader_uuid: "2512db3e6009416dbbcd437baf39d0f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2512db3e6009416dbbcd437baf39d0f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46257 } } }
I20260926 16:36:12.706657 27531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2512db3e6009416dbbcd437baf39d0f6 [sys.catalog]: This master's current role is: LEADER
I20260926 16:36:12.707197 27530 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2512db3e6009416dbbcd437baf39d0f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2512db3e6009416dbbcd437baf39d0f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2512db3e6009416dbbcd437baf39d0f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46257 } } }
I20260926 16:36:12.707494 27530 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2512db3e6009416dbbcd437baf39d0f6 [sys.catalog]: This master's current role is: LEADER
I20260926 16:36:12.710064 27536 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:36:12.717487 27536 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:36:12.726194 27536 catalog_manager.cc:1347] Generated new cluster ID: 87c1d00c39f0409e8bcf736278dab167
I20260926 16:36:12.726342 27536 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:36:12.738126 27536 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:36:12.740067 27536 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:36:12.753497 27536 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2512db3e6009416dbbcd437baf39d0f6: Generated new TSK 0
I20260926 16:36:12.754640 27536 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:36:12.760308 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 27452
W20260926 16:36:12.761478 27514 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.235:49347
W20260926 16:36:12.762332 6276 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 27452: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 16:36:12.762492 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 27452
W20260926 16:36:12.779703 27437 connection.cc:535] client connection to 127.0.0.1:46257 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46257): Connection reset by peer (error 104)
2026-09-26T16:36:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1941 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440233850748-6276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:36:12 dist-test-slave-khd3 krb5kdc[27550](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:36:12 dist-test-slave-khd3 krb5kdc[27550](info): set up 2 sockets
Sep 26 16:36:12 dist-test-slave-khd3 krb5kdc[27550](info): commencing operation
krb5kdc: starting...
W20260926 16:36:13.886205 6276 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Sep 26 16:36:14 dist-test-slave-khd3 krb5kdc[27550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:36:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:36:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440233850748-6276-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440233850748-6276-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 16:36:14.148434 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440233850748-6276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440233850748-6276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440233850748-6276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43953
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440233850748-6276-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43953
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:36:14.484311 27563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:36:14.484597 27563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:36:14.484800 27563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:36:14.492213 27563 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:36:14.492341 27563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:36:14.492450 27563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:36:14.492560 27563 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:36:14.492635 27563 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:36:14.503166 27563 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440233850748-6276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440233850748-6276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43953
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440233850748-6276-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440233850748-6276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:36:14.504551 27563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:36:14.505945 27563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:36:14.518041 27568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:36:14.518091 27571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:36:14.517987 27569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:36:14.519064 27563 server_base.cc:1034] running on GCE node
I20260926 16:36:14.519843 27563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:36:14.521399 27563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:36:14.522624 27563 hybrid_clock.cc:648] HybridClock initialized: now 1790440574522535 us; error 74 us; skew 500 ppm
I20260926 16:36:14.522993 27563 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:36:14 dist-test-slave-khd3 krb5kdc[27550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440574, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:36:14.526353 27563 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 16:36:14.527920 27563 webserver.cc:460] Webserver started at http://127.0.0.1:37155/ using document root <none> and password file <none>
I20260926 16:36:14.528679 27563 fs_manager.cc:359] Metadata directory not provided
I20260926 16:36:14.528879 27563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:36:14.529253 27563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:36:14.532214 27563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440233850748-6276-0/minicluster-data/master-0/data/instance:
uuid: "48db80e7b0474e4d84e01224b3156715"
format_stamp: "Formatted at 2026-09-26 16:36:14 on dist-test-slave-khd3"
server_key: "9bb1da8f59361193447d6d7380ca7bb1"
server_key_iv: "a5a785b80dc20a440a004aff639aec8c"
server_key_version: "encryptionkey@0"
I20260926 16:36:14.533069 27563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440233850748-6276-0/minicluster-data/master-0/wal/instance:
uuid: "48db80e7b0474e4d84e01224b3156715"
format_stamp: "Formatted at 2026-09-26 16:36:14 on dist-test-slave-khd3"
server_key: "9bb1da8f59361193447d6d7380ca7bb1"
server_key_iv: "a5a785b80dc20a440a004aff639aec8c"
server_key_version: "encryptionkey@0"
I20260926 16:36:14.538499 27563 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 16:36:14.542269 27578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:36:14.543910 27563 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:36:14.544209 27563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440233850748-6276-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440233850748-6276-0/minicluster-data/master-0/wal
uuid: "48db80e7b0474e4d84e01224b3156715"
format_stamp: "Formatted at 2026-09-26 16:36:14 on dist-test-slave-khd3"
server_key: "9bb1da8f59361193447d6d7380ca7bb1"
server_key_iv: "a5a785b80dc20a440a004aff639aec8c"
server_key_version: "encryptionkey@0"
I20260926 16:36:14.544435 27563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440233850748-6276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440233850748-6276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440233850748-6276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:36:14.550663 27563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:36:14.555527 27563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:36:14.555855 27563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:36:14.574978 27563 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43953
I20260926 16:36:14.575074 27629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43953 every 8 connection(s)
I20260926 16:36:14.576450 27563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
I20260926 16:36:14.580472 27630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:36:14.584929 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 27563
I20260926 16:36:14.585176 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440233850748-6276-0/minicluster-data/master-0/wal/instance
I20260926 16:36:14.585732 6276 external_mini_cluster.cc:1428] Setting key b19bf0a5731c3bb96e574759aae0519b
I20260926 16:36:14.593695 27630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48db80e7b0474e4d84e01224b3156715: Bootstrap starting.
I20260926 16:36:14.596072 27633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:14.587906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.235:57561 (local address 127.0.0.1:43953)
0926 16:36:14.588921 (+ 1015us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:14.588938 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:14.589309 (+ 371us) server_negotiation.cc:407] Connection header received
0926 16:36:14.591923 (+ 2614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:14.591963 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:14.592485 (+ 522us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:14.593009 (+ 524us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:14.594059 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:14.594067 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:14.594085 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:36:14.594109 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:14.594180 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:36:14.594207 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:14.594426 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:14.594553 (+ 127us) server_negotiation.cc:299] Negotiation successful
0926 16:36:14.594744 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":715,"thread_start_us":436,"threads_started":1}
I20260926 16:36:14.597618 27630 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48db80e7b0474e4d84e01224b3156715: Neither blocks nor log segments found. Creating new log.
I20260926 16:36:14.598783 27630 log.cc:824] T 00000000000000000000000000000000 P 48db80e7b0474e4d84e01224b3156715: Log is configured to *not* fsync() on all Append() calls
I20260926 16:36:14.600189 6276 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:36:14.601828 27630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48db80e7b0474e4d84e01224b3156715: No bootstrap required, opened a new log
I20260926 16:36:14.607473 27633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:14.604815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.235:55133 (local address 127.0.0.1:43953)
0926 16:36:14.604975 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:14.604979 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:14.605410 (+ 431us) server_negotiation.cc:407] Connection header received
0926 16:36:14.605551 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:14.605557 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:14.605628 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:14.605736 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:14.606770 (+ 1034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:14.606776 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:14.606781 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:36:14.606800 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:14.606855 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:36:14.606880 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:14.607022 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:14.607070 (+ 48us) server_negotiation.cc:299] Negotiation successful
0926 16:36:14.607137 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 16:36:14.611595 27630 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48db80e7b0474e4d84e01224b3156715 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48db80e7b0474e4d84e01224b3156715" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43953 } }
I20260926 16:36:14.611986 27630 raft_consensus.cc:374] T 00000000000000000000000000000000 P 48db80e7b0474e4d84e01224b3156715 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:36:14.612061 27630 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48db80e7b0474e4d84e01224b3156715 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48db80e7b0474e4d84e01224b3156715, State: Initialized, Role: FOLLOWER
I20260926 16:36:14.612661 27630 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48db80e7b0474e4d84e01224b3156715 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48db80e7b0474e4d84e01224b3156715" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43953 } }
I20260926 16:36:14.612932 27630 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48db80e7b0474e4d84e01224b3156715 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:36:14.613044 27630 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48db80e7b0474e4d84e01224b3156715 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:36:14.613175 27630 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48db80e7b0474e4d84e01224b3156715 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:36:14.615420 27630 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48db80e7b0474e4d84e01224b3156715 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48db80e7b0474e4d84e01224b3156715" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43953 } }
I20260926 16:36:14.615919 27630 leader_election.cc:302] T 00000000000000000000000000000000 P 48db80e7b0474e4d84e01224b3156715 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48db80e7b0474e4d84e01224b3156715; no voters:
I20260926 16:36:14.616518 27630 leader_election.cc:288] T 00000000000000000000000000000000 P 48db80e7b0474e4d84e01224b3156715 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:36:14.616712 27640 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48db80e7b0474e4d84e01224b3156715 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:36:14.617506 27640 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48db80e7b0474e4d84e01224b3156715 [term 1 LEADER]: Becoming Leader. State: Replica: 48db80e7b0474e4d84e01224b3156715, State: Running, Role: LEADER
I20260926 16:36:14.618149 27640 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48db80e7b0474e4d84e01224b3156715 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48db80e7b0474e4d84e01224b3156715" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43953 } }
I20260926 16:36:14.619073 27630 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48db80e7b0474e4d84e01224b3156715 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:36:14.621516 27641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48db80e7b0474e4d84e01224b3156715 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "48db80e7b0474e4d84e01224b3156715" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48db80e7b0474e4d84e01224b3156715" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43953 } } }
I20260926 16:36:14.622038 27641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48db80e7b0474e4d84e01224b3156715 [sys.catalog]: This master's current role is: LEADER
I20260926 16:36:14.622941 27642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48db80e7b0474e4d84e01224b3156715 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48db80e7b0474e4d84e01224b3156715. Latest consensus state: current_term: 1 leader_uuid: "48db80e7b0474e4d84e01224b3156715" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48db80e7b0474e4d84e01224b3156715" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43953 } } }
I20260926 16:36:14.623437 27642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48db80e7b0474e4d84e01224b3156715 [sys.catalog]: This master's current role is: LEADER
I20260926 16:36:14.624567 27648 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:36:14.630407 27648 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:36:14.639920 27648 catalog_manager.cc:1347] Generated new cluster ID: 1f58efc85c984f59bfbb9ba5380ceba7
I20260926 16:36:14.640091 27648 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:36:14.650053 27648 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:36:14.650861 27648 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:36:14.656785 27648 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 48db80e7b0474e4d84e01224b3156715: Generated new TSK 0
I20260926 16:36:14.657835 27648 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:36:14.663868 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 27563
W20260926 16:36:14.665210 27625 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.235:57561
W20260926 16:36:14.665902 6276 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 27563: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 16:36:14.666057 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 27563
2026-09-26T16:36:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1903 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440233850748-6276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:36:14 dist-test-slave-khd3 krb5kdc[27661](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:36:14 dist-test-slave-khd3 krb5kdc[27661](info): set up 2 sockets
Sep 26 16:36:14 dist-test-slave-khd3 krb5kdc[27661](info): commencing operation
krb5kdc: starting...
W20260926 16:36:15.782523 6276 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.002s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Sep 26 16:36:15 dist-test-slave-khd3 krb5kdc[27661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:36:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:36:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440233850748-6276-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440233850748-6276-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 16:36:16.047811 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440233850748-6276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440233850748-6276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440233850748-6276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41055
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440233850748-6276-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41055
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:36:16.386803 27674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:36:16.387163 27674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:36:16.387336 27674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:36:16.394452 27674 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:36:16.394618 27674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:36:16.394728 27674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:36:16.394841 27674 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:36:16.394930 27674 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:36:16.405434 27674 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440233850748-6276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440233850748-6276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41055
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440233850748-6276-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440233850748-6276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:36:16.406816 27674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:36:16.408169 27674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:36:16.420075 27679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:36:16.420109 27682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:36:16.420042 27680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:36:16.421060 27674 server_base.cc:1034] running on GCE node
I20260926 16:36:16.421911 27674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:36:16.423306 27674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:36:16.424549 27674 hybrid_clock.cc:648] HybridClock initialized: now 1790440576424499 us; error 49 us; skew 500 ppm
I20260926 16:36:16.424952 27674 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:36:16 dist-test-slave-khd3 krb5kdc[27661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440576, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:36:16.428231 27674 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 16:36:16.430290 27674 webserver.cc:460] Webserver started at http://127.0.0.1:38997/ using document root <none> and password file <none>
I20260926 16:36:16.431033 27674 fs_manager.cc:359] Metadata directory not provided
I20260926 16:36:16.431195 27674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:36:16.431505 27674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:36:16.434643 27674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440233850748-6276-0/minicluster-data/master-0/data/instance:
uuid: "5242e62bb06e427ba4ec89cb77f3ea64"
format_stamp: "Formatted at 2026-09-26 16:36:16 on dist-test-slave-khd3"
server_key: "9d38066037aa187822fb76f302b95720"
server_key_iv: "7ad6b8bab27c08776d8b7709e7d3337e"
server_key_version: "encryptionkey@0"
I20260926 16:36:16.435410 27674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440233850748-6276-0/minicluster-data/master-0/wal/instance:
uuid: "5242e62bb06e427ba4ec89cb77f3ea64"
format_stamp: "Formatted at 2026-09-26 16:36:16 on dist-test-slave-khd3"
server_key: "9d38066037aa187822fb76f302b95720"
server_key_iv: "7ad6b8bab27c08776d8b7709e7d3337e"
server_key_version: "encryptionkey@0"
I20260926 16:36:16.440922 27674 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 16:36:16.444515 27689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:36:16.445999 27674 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:36:16.446198 27674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440233850748-6276-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440233850748-6276-0/minicluster-data/master-0/wal
uuid: "5242e62bb06e427ba4ec89cb77f3ea64"
format_stamp: "Formatted at 2026-09-26 16:36:16 on dist-test-slave-khd3"
server_key: "9d38066037aa187822fb76f302b95720"
server_key_iv: "7ad6b8bab27c08776d8b7709e7d3337e"
server_key_version: "encryptionkey@0"
I20260926 16:36:16.446405 27674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440233850748-6276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440233850748-6276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440233850748-6276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:36:16.464569 27674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:36:16.469350 27674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:36:16.469660 27674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:36:16.488865 27674 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41055
I20260926 16:36:16.488885 27740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41055 every 8 connection(s)
I20260926 16:36:16.490252 27674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
I20260926 16:36:16.493160 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 27674
I20260926 16:36:16.493358 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440233850748-6276-0/minicluster-data/master-0/wal/instance
I20260926 16:36:16.493789 6276 external_mini_cluster.cc:1428] Setting key b7122c4a1d80325208d15cd928937d0a
I20260926 16:36:16.494349 27741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 16:36:16 dist-test-slave-khd3 krb5kdc[27661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 16:36:16.509486 27741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5242e62bb06e427ba4ec89cb77f3ea64: Bootstrap starting.
I20260926 16:36:16.513957 27741 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5242e62bb06e427ba4ec89cb77f3ea64: Neither blocks nor log segments found. Creating new log.
I20260926 16:36:16.515267 27741 log.cc:824] T 00000000000000000000000000000000 P 5242e62bb06e427ba4ec89cb77f3ea64: Log is configured to *not* fsync() on all Append() calls
I20260926 16:36:16.518395 27744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:16.495554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.235:38635 (local address 127.0.0.1:41055)
0926 16:36:16.496283 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:16.496295 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:16.496336 (+ 41us) server_negotiation.cc:407] Connection header received
0926 16:36:16.497940 (+ 1604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:16.497961 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:16.498412 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:16.498749 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:16.500609 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:16.501997 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:16.503800 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:16.504444 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:16.504568 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:16.509458 (+ 4890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:16.509477 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:16.509497 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:36:16.509548 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:16.513607 (+ 4059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:16.514535 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:16.514544 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:16.514552 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:16.514765 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:16.515199 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:16.515207 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:16.515213 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:16.516014 (+ 801us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:16.516369 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:16.517022 (+ 653us) server_negotiation.cc:299] Negotiation successful
0926 16:36:16.517387 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":323,"threads_started":1}
I20260926 16:36:16.518693 27741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5242e62bb06e427ba4ec89cb77f3ea64: No bootstrap required, opened a new log
I20260926 16:36:16.522639 6276 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:36:16.529156 27741 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5242e62bb06e427ba4ec89cb77f3ea64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5242e62bb06e427ba4ec89cb77f3ea64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41055 } }
I20260926 16:36:16.529559 27741 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5242e62bb06e427ba4ec89cb77f3ea64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:36:16.529670 27741 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5242e62bb06e427ba4ec89cb77f3ea64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5242e62bb06e427ba4ec89cb77f3ea64, State: Initialized, Role: FOLLOWER
I20260926 16:36:16.530315 27741 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5242e62bb06e427ba4ec89cb77f3ea64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5242e62bb06e427ba4ec89cb77f3ea64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41055 } }
I20260926 16:36:16.530561 27741 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5242e62bb06e427ba4ec89cb77f3ea64 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:36:16.530704 27741 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5242e62bb06e427ba4ec89cb77f3ea64 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:36:16.530894 27741 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5242e62bb06e427ba4ec89cb77f3ea64 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:36:16.533277 27741 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5242e62bb06e427ba4ec89cb77f3ea64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5242e62bb06e427ba4ec89cb77f3ea64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41055 } }
I20260926 16:36:16.533815 27741 leader_election.cc:302] T 00000000000000000000000000000000 P 5242e62bb06e427ba4ec89cb77f3ea64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5242e62bb06e427ba4ec89cb77f3ea64; no voters:
I20260926 16:36:16.534426 27741 leader_election.cc:288] T 00000000000000000000000000000000 P 5242e62bb06e427ba4ec89cb77f3ea64 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:36:16.534605 27751 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5242e62bb06e427ba4ec89cb77f3ea64 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:36:16.535746 27751 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5242e62bb06e427ba4ec89cb77f3ea64 [term 1 LEADER]: Becoming Leader. State: Replica: 5242e62bb06e427ba4ec89cb77f3ea64, State: Running, Role: LEADER
I20260926 16:36:16.536470 27751 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5242e62bb06e427ba4ec89cb77f3ea64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5242e62bb06e427ba4ec89cb77f3ea64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41055 } }
I20260926 16:36:16.536681 27741 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5242e62bb06e427ba4ec89cb77f3ea64 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:36:16.539392 27753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5242e62bb06e427ba4ec89cb77f3ea64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5242e62bb06e427ba4ec89cb77f3ea64. Latest consensus state: current_term: 1 leader_uuid: "5242e62bb06e427ba4ec89cb77f3ea64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5242e62bb06e427ba4ec89cb77f3ea64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41055 } } }
I20260926 16:36:16.539778 27753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5242e62bb06e427ba4ec89cb77f3ea64 [sys.catalog]: This master's current role is: LEADER
I20260926 16:36:16.540318 27752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5242e62bb06e427ba4ec89cb77f3ea64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5242e62bb06e427ba4ec89cb77f3ea64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5242e62bb06e427ba4ec89cb77f3ea64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41055 } } }
I20260926 16:36:16.540675 27752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5242e62bb06e427ba4ec89cb77f3ea64 [sys.catalog]: This master's current role is: LEADER
I20260926 16:36:16.544019 27756 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:36:16.547716 27744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:16.528078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.235:58435 (local address 127.0.0.1:41055)
0926 16:36:16.528333 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:16.528342 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:16.528967 (+ 625us) server_negotiation.cc:407] Connection header received
0926 16:36:16.529181 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:16.529185 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:16.529252 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:16.529357 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:16.530930 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:16.531790 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:16.533194 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:16.533732 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:16.533893 (+ 161us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:16.537339 (+ 3446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:16.537345 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:16.537349 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:36:16.537387 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:16.542361 (+ 4974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:16.543558 (+ 1197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:16.543565 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:16.543569 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:16.543760 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:16.544466 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:16.544475 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:16.544481 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:16.545081 (+ 600us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:16.545463 (+ 382us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:16.547103 (+ 1640us) server_negotiation.cc:299] Negotiation successful
0926 16:36:16.547315 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":70}
I20260926 16:36:16.552742 27756 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:36:16.563791 27756 catalog_manager.cc:1347] Generated new cluster ID: 410655ecebf64704b55c52605761b256
I20260926 16:36:16.563921 27756 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:36:16.592849 27756 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:36:16.594702 27756 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:36:16.606696 27756 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5242e62bb06e427ba4ec89cb77f3ea64: Generated new TSK 0
I20260926 16:36:16.607806 27756 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:36:16.620441 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 27674
I20260926 16:36:16.621932 27736 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:36:16.717700 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 27674
W20260926 16:36:16.734788 27659 connection.cc:535] client connection to 127.0.0.1:41055 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41055): Connection reset by peer (error 104)
2026-09-26T16:36:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2051 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440233850748-6276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:36:16 dist-test-slave-khd3 krb5kdc[27774](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:36:16 dist-test-slave-khd3 krb5kdc[27774](info): set up 2 sockets
Sep 26 16:36:16 dist-test-slave-khd3 krb5kdc[27774](info): commencing operation
krb5kdc: starting...
W20260926 16:36:17.822238 6276 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Sep 26 16:36:17 dist-test-slave-khd3 krb5kdc[27774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:36:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:36:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440233850748-6276-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440233850748-6276-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 16:36:18.096650 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440233850748-6276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440233850748-6276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440233850748-6276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42359
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440233850748-6276-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42359
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:36:18.437546 27787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:36:18.437872 27787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:36:18.438051 27787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:36:18.445729 27787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:36:18.445866 27787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:36:18.445953 27787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:36:18.446054 27787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:36:18.446169 27787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:36:18.456461 27787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440233850748-6276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440233850748-6276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42359
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440233850748-6276-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440233850748-6276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:36:18.457923 27787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:36:18.459344 27787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:36:18.471859 27792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:36:18.471884 27793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:36:18.471812 27795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:36:18.472709 27787 server_base.cc:1034] running on GCE node
I20260926 16:36:18.473584 27787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:36:18.475035 27787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:36:18.476265 27787 hybrid_clock.cc:648] HybridClock initialized: now 1790440578476195 us; error 58 us; skew 500 ppm
I20260926 16:36:18.476660 27787 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:36:18 dist-test-slave-khd3 krb5kdc[27774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440578, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:36:18.480099 27787 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 16:36:18.481729 27787 webserver.cc:460] Webserver started at http://127.0.0.1:36351/ using document root <none> and password file <none>
I20260926 16:36:18.482448 27787 fs_manager.cc:359] Metadata directory not provided
I20260926 16:36:18.482585 27787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:36:18.482915 27787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:36:18.486075 27787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440233850748-6276-0/minicluster-data/master-0/data/instance:
uuid: "461540028b17486ba3b2b72b36539688"
format_stamp: "Formatted at 2026-09-26 16:36:18 on dist-test-slave-khd3"
server_key: "f320145faa43b81765139e121e0f1422"
server_key_iv: "2acd4edb3c51cdfa86f502db429afa1b"
server_key_version: "encryptionkey@0"
I20260926 16:36:18.486891 27787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440233850748-6276-0/minicluster-data/master-0/wal/instance:
uuid: "461540028b17486ba3b2b72b36539688"
format_stamp: "Formatted at 2026-09-26 16:36:18 on dist-test-slave-khd3"
server_key: "f320145faa43b81765139e121e0f1422"
server_key_iv: "2acd4edb3c51cdfa86f502db429afa1b"
server_key_version: "encryptionkey@0"
I20260926 16:36:18.492815 27787 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 16:36:18.496404 27802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:36:18.497952 27787 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 16:36:18.498252 27787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440233850748-6276-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440233850748-6276-0/minicluster-data/master-0/wal
uuid: "461540028b17486ba3b2b72b36539688"
format_stamp: "Formatted at 2026-09-26 16:36:18 on dist-test-slave-khd3"
server_key: "f320145faa43b81765139e121e0f1422"
server_key_iv: "2acd4edb3c51cdfa86f502db429afa1b"
server_key_version: "encryptionkey@0"
I20260926 16:36:18.498466 27787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440233850748-6276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440233850748-6276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440233850748-6276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:36:18.522971 27787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:36:18.527447 27787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:36:18.527770 27787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:36:18.547595 27787 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42359
I20260926 16:36:18.547643 27853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42359 every 8 connection(s)
I20260926 16:36:18.549075 27787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
I20260926 16:36:18.553143 27854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:36:18.553224 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 27787
I20260926 16:36:18.553428 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440233850748-6276-0/minicluster-data/master-0/wal/instance
I20260926 16:36:18.553936 6276 external_mini_cluster.cc:1428] Setting key d90a3e758069923d4f39b43834253e08
I20260926 16:36:18.569386 27854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 461540028b17486ba3b2b72b36539688: Bootstrap starting.
I20260926 16:36:18.569962 27857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:18.555881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.235:49623 (local address 127.0.0.1:42359)
0926 16:36:18.556733 (+ 852us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:18.556749 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:18.556827 (+ 78us) server_negotiation.cc:407] Connection header received
0926 16:36:18.558733 (+ 1906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:18.558772 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:18.559393 (+ 621us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:18.559964 (+ 571us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:18.560543 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:18.562705 (+ 2162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:18.564176 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:18.565355 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:18.565542 (+ 187us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:18.566812 (+ 1270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:18.566826 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:18.566852 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:36:18.567625 (+ 773us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:18.567735 (+ 110us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:36:18.567778 (+ 43us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:18.567931 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:18.568261 (+ 330us) server_negotiation.cc:299] Negotiation successful
0926 16:36:18.568465 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":357,"threads_started":1}
I20260926 16:36:18.572948 27854 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 461540028b17486ba3b2b72b36539688: Neither blocks nor log segments found. Creating new log.
I20260926 16:36:18.574245 27854 log.cc:824] T 00000000000000000000000000000000 P 461540028b17486ba3b2b72b36539688: Log is configured to *not* fsync() on all Append() calls
I20260926 16:36:18.574267 6276 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:36:18.577729 27854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 461540028b17486ba3b2b72b36539688: No bootstrap required, opened a new log
I20260926 16:36:18.587093 27857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:18.579542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.235:60895 (local address 127.0.0.1:42359)
0926 16:36:18.579761 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:18.579771 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:18.580313 (+ 542us) server_negotiation.cc:407] Connection header received
0926 16:36:18.580479 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:18.580485 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:18.580632 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:18.580791 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:18.581386 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:18.582546 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:18.583906 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:18.584642 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:18.584805 (+ 163us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:18.586213 (+ 1408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:18.586220 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:18.586225 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:36:18.586254 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:18.586314 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:36:18.586336 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:18.586449 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:18.586681 (+ 232us) server_negotiation.cc:299] Negotiation successful
0926 16:36:18.586739 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 16:36:18.588020 27854 raft_consensus.cc:348] T 00000000000000000000000000000000 P 461540028b17486ba3b2b72b36539688 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "461540028b17486ba3b2b72b36539688" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42359 } }
I20260926 16:36:18.588450 27854 raft_consensus.cc:374] T 00000000000000000000000000000000 P 461540028b17486ba3b2b72b36539688 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:36:18.588572 27854 raft_consensus.cc:729] T 00000000000000000000000000000000 P 461540028b17486ba3b2b72b36539688 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 461540028b17486ba3b2b72b36539688, State: Initialized, Role: FOLLOWER
I20260926 16:36:18.589253 27854 consensus_queue.cc:260] T 00000000000000000000000000000000 P 461540028b17486ba3b2b72b36539688 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "461540028b17486ba3b2b72b36539688" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42359 } }
I20260926 16:36:18.589512 27854 raft_consensus.cc:388] T 00000000000000000000000000000000 P 461540028b17486ba3b2b72b36539688 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:36:18.589654 27854 raft_consensus.cc:482] T 00000000000000000000000000000000 P 461540028b17486ba3b2b72b36539688 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:36:18.589843 27854 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 461540028b17486ba3b2b72b36539688 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:36:18.592059 27854 raft_consensus.cc:504] T 00000000000000000000000000000000 P 461540028b17486ba3b2b72b36539688 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "461540028b17486ba3b2b72b36539688" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42359 } }
I20260926 16:36:18.592608 27854 leader_election.cc:302] T 00000000000000000000000000000000 P 461540028b17486ba3b2b72b36539688 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 461540028b17486ba3b2b72b36539688; no voters:
I20260926 16:36:18.593268 27854 leader_election.cc:288] T 00000000000000000000000000000000 P 461540028b17486ba3b2b72b36539688 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:36:18.593413 27864 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 461540028b17486ba3b2b72b36539688 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:36:18.594652 27864 raft_consensus.cc:686] T 00000000000000000000000000000000 P 461540028b17486ba3b2b72b36539688 [term 1 LEADER]: Becoming Leader. State: Replica: 461540028b17486ba3b2b72b36539688, State: Running, Role: LEADER
I20260926 16:36:18.595235 27864 consensus_queue.cc:237] T 00000000000000000000000000000000 P 461540028b17486ba3b2b72b36539688 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "461540028b17486ba3b2b72b36539688" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42359 } }
I20260926 16:36:18.595671 27854 sys_catalog.cc:564] T 00000000000000000000000000000000 P 461540028b17486ba3b2b72b36539688 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:36:18.598371 27865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 461540028b17486ba3b2b72b36539688 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "461540028b17486ba3b2b72b36539688" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "461540028b17486ba3b2b72b36539688" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42359 } } }
I20260926 16:36:18.598755 27865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 461540028b17486ba3b2b72b36539688 [sys.catalog]: This master's current role is: LEADER
I20260926 16:36:18.598455 27866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 461540028b17486ba3b2b72b36539688 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 461540028b17486ba3b2b72b36539688. Latest consensus state: current_term: 1 leader_uuid: "461540028b17486ba3b2b72b36539688" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "461540028b17486ba3b2b72b36539688" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42359 } } }
I20260926 16:36:18.598887 27866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 461540028b17486ba3b2b72b36539688 [sys.catalog]: This master's current role is: LEADER
I20260926 16:36:18.605360 27872 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:36:18.610932 27872 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:36:18.621068 27872 catalog_manager.cc:1347] Generated new cluster ID: 81b8d26d74154509ad8fe38d986f677b
I20260926 16:36:18.621201 27872 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:36:18.639425 27872 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:36:18.640851 27872 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:36:18.648177 27872 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 461540028b17486ba3b2b72b36539688: Generated new TSK 0
I20260926 16:36:18.649140 27872 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:36:18.661331 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 27787
W20260926 16:36:18.663625 27849 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.235:49623
W20260926 16:36:18.664448 6276 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 27787: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 16:36:18.664639 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 27787
W20260926 16:36:18.681720 27772 connection.cc:535] client connection to 127.0.0.1:42359 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42359): Connection reset by peer (error 104)
2026-09-26T16:36:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1947 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440233850748-6276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:36:18 dist-test-slave-khd3 krb5kdc[27885](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:36:18 dist-test-slave-khd3 krb5kdc[27885](info): set up 2 sockets
Sep 26 16:36:18 dist-test-slave-khd3 krb5kdc[27885](info): commencing operation
krb5kdc: starting...
W20260926 16:36:19.798317 6276 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.004s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Sep 26 16:36:19 dist-test-slave-khd3 krb5kdc[27885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:36:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:36:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440233850748-6276-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440233850748-6276-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 16:36:20.064270 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440233850748-6276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440233850748-6276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440233850748-6276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45759
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440233850748-6276-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45759
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:36:20.402271 27898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:36:20.402626 27898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:36:20.402806 27898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:36:20.410012 27898 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:36:20.410132 27898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:36:20.410223 27898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:36:20.410321 27898 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:36:20.410410 27898 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:36:20.420280 27898 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440233850748-6276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440233850748-6276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45759
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440233850748-6276-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440233850748-6276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:36:20.421660 27898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:36:20.423004 27898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:36:20.436455 27906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:36:20.436506 27903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:36:20.436470 27904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:36:20.437227 27898 server_base.cc:1034] running on GCE node
I20260926 16:36:20.438089 27898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:36:20.439503 27898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:36:20.440717 27898 hybrid_clock.cc:648] HybridClock initialized: now 1790440580440686 us; error 34 us; skew 500 ppm
I20260926 16:36:20.441133 27898 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:36:20 dist-test-slave-khd3 krb5kdc[27885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440580, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:36:20.444388 27898 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 16:36:20.446038 27898 webserver.cc:460] Webserver started at http://127.0.0.1:43193/ using document root <none> and password file <none>
I20260926 16:36:20.446753 27898 fs_manager.cc:359] Metadata directory not provided
I20260926 16:36:20.446899 27898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:36:20.447206 27898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:36:20.450279 27898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440233850748-6276-0/minicluster-data/master-0/data/instance:
uuid: "b5e325e5251944d1872d06db613b6ff1"
format_stamp: "Formatted at 2026-09-26 16:36:20 on dist-test-slave-khd3"
server_key: "1c6c7b0d312eb873692410bedcbf8f54"
server_key_iv: "448a0b2b4a69df37b1b2331e4c951ca6"
server_key_version: "encryptionkey@0"
I20260926 16:36:20.451099 27898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440233850748-6276-0/minicluster-data/master-0/wal/instance:
uuid: "b5e325e5251944d1872d06db613b6ff1"
format_stamp: "Formatted at 2026-09-26 16:36:20 on dist-test-slave-khd3"
server_key: "1c6c7b0d312eb873692410bedcbf8f54"
server_key_iv: "448a0b2b4a69df37b1b2331e4c951ca6"
server_key_version: "encryptionkey@0"
I20260926 16:36:20.456432 27898 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260926 16:36:20.460234 27913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:36:20.461694 27898 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:36:20.461892 27898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440233850748-6276-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440233850748-6276-0/minicluster-data/master-0/wal
uuid: "b5e325e5251944d1872d06db613b6ff1"
format_stamp: "Formatted at 2026-09-26 16:36:20 on dist-test-slave-khd3"
server_key: "1c6c7b0d312eb873692410bedcbf8f54"
server_key_iv: "448a0b2b4a69df37b1b2331e4c951ca6"
server_key_version: "encryptionkey@0"
I20260926 16:36:20.462113 27898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440233850748-6276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440233850748-6276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440233850748-6276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:36:20.468129 27898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:36:20.472470 27898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:36:20.472813 27898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:36:20.490794 27898 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45759
I20260926 16:36:20.490839 27964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45759 every 8 connection(s)
I20260926 16:36:20.492239 27898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
I20260926 16:36:20.496114 27965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:36:20.500059 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 27898
I20260926 16:36:20.500249 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440233850748-6276-0/minicluster-data/master-0/wal/instance
I20260926 16:36:20.500818 6276 external_mini_cluster.cc:1428] Setting key 364651271b049259430e3a94f695a57e
I20260926 16:36:20.510331 27968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:20.502646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.235:48229 (local address 127.0.0.1:45759)
0926 16:36:20.503662 (+ 1016us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:20.503682 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:20.503741 (+ 59us) server_negotiation.cc:407] Connection header received
0926 16:36:20.505973 (+ 2232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:20.506006 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:20.506547 (+ 541us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:20.507045 (+ 498us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:20.508162 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:20.508172 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:20.508196 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:36:20.508224 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:20.508296 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:36:20.508324 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:20.508599 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:20.508749 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 16:36:20.508995 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":736,"thread_start_us":468,"threads_started":1}
I20260926 16:36:20.511308 27965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5e325e5251944d1872d06db613b6ff1: Bootstrap starting.
I20260926 16:36:20.514336 6276 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:36:20.515110 27965 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5e325e5251944d1872d06db613b6ff1: Neither blocks nor log segments found. Creating new log.
I20260926 16:36:20.516218 27965 log.cc:824] T 00000000000000000000000000000000 P b5e325e5251944d1872d06db613b6ff1: Log is configured to *not* fsync() on all Append() calls
I20260926 16:36:20.519770 27965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5e325e5251944d1872d06db613b6ff1: No bootstrap required, opened a new log
I20260926 16:36:20.521456 27968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:20.518750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.235:41021 (local address 127.0.0.1:45759)
0926 16:36:20.518951 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:20.518960 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:20.519437 (+ 477us) server_negotiation.cc:407] Connection header received
0926 16:36:20.519570 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:20.519578 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:20.519638 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:20.519767 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:20.520716 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:20.520733 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:20.520738 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:36:20.520810 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:20.520862 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:36:20.520887 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:20.521040 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:20.521092 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 16:36:20.521155 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 16:36:20.528180 27965 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5e325e5251944d1872d06db613b6ff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5e325e5251944d1872d06db613b6ff1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45759 } }
I20260926 16:36:20.528617 27965 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5e325e5251944d1872d06db613b6ff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:36:20.528738 27965 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5e325e5251944d1872d06db613b6ff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5e325e5251944d1872d06db613b6ff1, State: Initialized, Role: FOLLOWER
I20260926 16:36:20.529387 27965 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5e325e5251944d1872d06db613b6ff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5e325e5251944d1872d06db613b6ff1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45759 } }
I20260926 16:36:20.529659 27965 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5e325e5251944d1872d06db613b6ff1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:36:20.529842 27965 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5e325e5251944d1872d06db613b6ff1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:36:20.530031 27965 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5e325e5251944d1872d06db613b6ff1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:36:20.532589 27965 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5e325e5251944d1872d06db613b6ff1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5e325e5251944d1872d06db613b6ff1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45759 } }
I20260926 16:36:20.533319 27965 leader_election.cc:302] T 00000000000000000000000000000000 P b5e325e5251944d1872d06db613b6ff1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5e325e5251944d1872d06db613b6ff1; no voters:
I20260926 16:36:20.534001 27965 leader_election.cc:288] T 00000000000000000000000000000000 P b5e325e5251944d1872d06db613b6ff1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:36:20.534180 27975 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5e325e5251944d1872d06db613b6ff1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:36:20.534891 27975 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5e325e5251944d1872d06db613b6ff1 [term 1 LEADER]: Becoming Leader. State: Replica: b5e325e5251944d1872d06db613b6ff1, State: Running, Role: LEADER
I20260926 16:36:20.535557 27975 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5e325e5251944d1872d06db613b6ff1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5e325e5251944d1872d06db613b6ff1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45759 } }
I20260926 16:36:20.536384 27965 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5e325e5251944d1872d06db613b6ff1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:36:20.538393 27977 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5e325e5251944d1872d06db613b6ff1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5e325e5251944d1872d06db613b6ff1. Latest consensus state: current_term: 1 leader_uuid: "b5e325e5251944d1872d06db613b6ff1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5e325e5251944d1872d06db613b6ff1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45759 } } }
I20260926 16:36:20.538919 27977 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5e325e5251944d1872d06db613b6ff1 [sys.catalog]: This master's current role is: LEADER
I20260926 16:36:20.538774 27976 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5e325e5251944d1872d06db613b6ff1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5e325e5251944d1872d06db613b6ff1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5e325e5251944d1872d06db613b6ff1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45759 } } }
I20260926 16:36:20.543200 27976 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5e325e5251944d1872d06db613b6ff1 [sys.catalog]: This master's current role is: LEADER
W20260926 16:36:20.548468 27991 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b5e325e5251944d1872d06db613b6ff1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 16:36:20.548597 27991 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 16:36:20.548746 27983 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:36:20.551039 27983 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:36:20.560877 27983 catalog_manager.cc:1347] Generated new cluster ID: ff8812bcf8d347c181010c121b693e80
I20260926 16:36:20.561014 27983 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:36:20.580547 27983 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:36:20.581095 27983 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:36:20.599857 27983 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5e325e5251944d1872d06db613b6ff1: Generated new TSK 0
I20260926 16:36:20.601183 27983 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:36:20.606545 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 27898
W20260926 16:36:20.607851 27960 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.235:48229
W20260926 16:36:20.608855 6276 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 27898: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 16:36:20.609081 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 27898
2026-09-26T16:36:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1944 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440233850748-6276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:36:20 dist-test-slave-khd3 krb5kdc[27996](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:36:20 dist-test-slave-khd3 krb5kdc[27996](info): set up 2 sockets
Sep 26 16:36:20 dist-test-slave-khd3 krb5kdc[27996](info): commencing operation
krb5kdc: starting...
W20260926 16:36:21.726163 6276 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Sep 26 16:36:21 dist-test-slave-khd3 krb5kdc[27996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:36:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:36:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab.
Entry for principal kudu/127.6.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab.
I20260926 16:36:21.992017 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440233850748-6276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440233850748-6276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440233850748-6276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:43789
--webserver_interface=127.6.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:38355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.33.62:43789
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:36:22.338680 28009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:36:22.339039 28009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:36:22.339205 28009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:36:22.346728 28009 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:36:22.346887 28009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:36:22.347013 28009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:36:22.347121 28009 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:36:22.347206 28009 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:36:22.357553 28009 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:38355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440233850748-6276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440233850748-6276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.33.62:43789
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:43789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440233850748-6276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:36:22.358985 28009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:36:22.360430 28009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:36:22.373158 28014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:36:22.373114 28017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:36:22.373181 28015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:36:22.374061 28009 server_base.cc:1034] running on GCE node
I20260926 16:36:22.374895 28009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:36:22.376427 28009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:36:22.377669 28009 hybrid_clock.cc:648] HybridClock initialized: now 1790440582377612 us; error 54 us; skew 500 ppm
I20260926 16:36:22.378046 28009 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:36:22 dist-test-slave-khd3 krb5kdc[27996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440582, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:36:22.381264 28009 init.cc:377] Logged in from keytab as kudu/127.6.33.62@KRBTEST.COM (short username kudu)
I20260926 16:36:22.382854 28009 webserver.cc:460] Webserver started at http://127.6.33.62:39465/ using document root <none> and password file <none>
I20260926 16:36:22.383560 28009 fs_manager.cc:359] Metadata directory not provided
I20260926 16:36:22.383703 28009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:36:22.384011 28009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:36:22.387200 28009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440233850748-6276-0/minicluster-data/master-0/data/instance:
uuid: "1e468515022a4d91bd4ca7b4aef6cc2f"
format_stamp: "Formatted at 2026-09-26 16:36:22 on dist-test-slave-khd3"
server_key: "bb9a235a5a95a104fb0e1040f85c774d"
server_key_iv: "3ee391788554729adb70255ee08018ce"
server_key_version: "encryptionkey@0"
I20260926 16:36:22.387972 28009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440233850748-6276-0/minicluster-data/master-0/wal/instance:
uuid: "1e468515022a4d91bd4ca7b4aef6cc2f"
format_stamp: "Formatted at 2026-09-26 16:36:22 on dist-test-slave-khd3"
server_key: "bb9a235a5a95a104fb0e1040f85c774d"
server_key_iv: "3ee391788554729adb70255ee08018ce"
server_key_version: "encryptionkey@0"
I20260926 16:36:22.393257 28009 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 16:36:22.396739 28024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:36:22.398106 28009 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:36:22.398293 28009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440233850748-6276-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440233850748-6276-0/minicluster-data/master-0/wal
uuid: "1e468515022a4d91bd4ca7b4aef6cc2f"
format_stamp: "Formatted at 2026-09-26 16:36:22 on dist-test-slave-khd3"
server_key: "bb9a235a5a95a104fb0e1040f85c774d"
server_key_iv: "3ee391788554729adb70255ee08018ce"
server_key_version: "encryptionkey@0"
I20260926 16:36:22.398486 28009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440233850748-6276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440233850748-6276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440233850748-6276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:36:22.411307 28009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:36:22.415843 28009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:36:22.416165 28009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:36:22.435490 28009 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.62:43789
I20260926 16:36:22.435519 28075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.62:43789 every 8 connection(s)
I20260926 16:36:22.437032 28009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
I20260926 16:36:22.437744 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 28009
I20260926 16:36:22.437954 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440233850748-6276-0/minicluster-data/master-0/wal/instance
I20260926 16:36:22.438589 6276 external_mini_cluster.cc:1428] Setting key 91b0097070bf8b2ed1243a6ad2765d67
I20260926 16:36:22.441789 28076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 16:36:22 dist-test-slave-khd3 krb5kdc[27996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:36:22.452996 28076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e468515022a4d91bd4ca7b4aef6cc2f: Bootstrap starting.
I20260926 16:36:22.457585 28076 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e468515022a4d91bd4ca7b4aef6cc2f: Neither blocks nor log segments found. Creating new log.
I20260926 16:36:22.458926 28076 log.cc:824] T 00000000000000000000000000000000 P 1e468515022a4d91bd4ca7b4aef6cc2f: Log is configured to *not* fsync() on all Append() calls
I20260926 16:36:22.462709 28076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e468515022a4d91bd4ca7b4aef6cc2f: No bootstrap required, opened a new log
I20260926 16:36:22.462673 28079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:22.440436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56305 (local address 127.6.33.62:43789)
0926 16:36:22.441281 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:22.441294 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:22.441334 (+ 40us) server_negotiation.cc:407] Connection header received
0926 16:36:22.442972 (+ 1638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:22.442994 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:22.443372 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:22.443741 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:22.445301 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:22.446672 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:22.448100 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:22.448858 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:22.449074 (+ 216us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:22.453452 (+ 4378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:22.453465 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:22.453487 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:36:22.453543 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:22.457754 (+ 4211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:22.458645 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:22.458657 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:22.458670 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:22.458888 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:22.459456 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:22.459463 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:22.459468 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:22.460111 (+ 643us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:22.460446 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:22.460959 (+ 513us) server_negotiation.cc:299] Negotiation successful
0926 16:36:22.461312 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":358,"threads_started":1}
I20260926 16:36:22.466809 6276 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:36:22.471361 28076 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e468515022a4d91bd4ca7b4aef6cc2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e468515022a4d91bd4ca7b4aef6cc2f" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 43789 } }
I20260926 16:36:22.471849 28076 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e468515022a4d91bd4ca7b4aef6cc2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:36:22.471974 28076 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e468515022a4d91bd4ca7b4aef6cc2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e468515022a4d91bd4ca7b4aef6cc2f, State: Initialized, Role: FOLLOWER
I20260926 16:36:22.472640 28076 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e468515022a4d91bd4ca7b4aef6cc2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e468515022a4d91bd4ca7b4aef6cc2f" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 43789 } }
I20260926 16:36:22.472929 28076 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e468515022a4d91bd4ca7b4aef6cc2f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:36:22.473090 28076 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e468515022a4d91bd4ca7b4aef6cc2f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:36:22.473287 28076 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e468515022a4d91bd4ca7b4aef6cc2f [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:36:22.476197 28076 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e468515022a4d91bd4ca7b4aef6cc2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e468515022a4d91bd4ca7b4aef6cc2f" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 43789 } }
I20260926 16:36:22.476879 28076 leader_election.cc:302] T 00000000000000000000000000000000 P 1e468515022a4d91bd4ca7b4aef6cc2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e468515022a4d91bd4ca7b4aef6cc2f; no voters:
I20260926 16:36:22.477609 28076 leader_election.cc:288] T 00000000000000000000000000000000 P 1e468515022a4d91bd4ca7b4aef6cc2f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:36:22.477711 28086 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e468515022a4d91bd4ca7b4aef6cc2f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:36:22.478914 28086 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e468515022a4d91bd4ca7b4aef6cc2f [term 1 LEADER]: Becoming Leader. State: Replica: 1e468515022a4d91bd4ca7b4aef6cc2f, State: Running, Role: LEADER
I20260926 16:36:22.479566 28086 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e468515022a4d91bd4ca7b4aef6cc2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e468515022a4d91bd4ca7b4aef6cc2f" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 43789 } }
I20260926 16:36:22.480152 28076 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e468515022a4d91bd4ca7b4aef6cc2f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:36:22.482831 28087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e468515022a4d91bd4ca7b4aef6cc2f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e468515022a4d91bd4ca7b4aef6cc2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e468515022a4d91bd4ca7b4aef6cc2f" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 43789 } } }
I20260926 16:36:22.483301 28087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e468515022a4d91bd4ca7b4aef6cc2f [sys.catalog]: This master's current role is: LEADER
I20260926 16:36:22.487393 28079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:22.471113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41689 (local address 127.6.33.62:43789)
0926 16:36:22.471290 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:22.471297 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:22.471636 (+ 339us) server_negotiation.cc:407] Connection header received
0926 16:36:22.471773 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:22.471778 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:22.471868 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:22.472067 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:22.473735 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:22.474965 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:22.476242 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:22.476994 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:22.477129 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:22.480022 (+ 2893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:22.480026 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:22.480030 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:36:22.480061 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:22.483844 (+ 3783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:22.484599 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:22.484604 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:22.484608 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:22.484817 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:22.485455 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:22.485463 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:22.485468 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:22.485925 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:22.486306 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:22.486664 (+ 358us) server_negotiation.cc:299] Negotiation successful
0926 16:36:22.486840 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 16:36:22.487754 28088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e468515022a4d91bd4ca7b4aef6cc2f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e468515022a4d91bd4ca7b4aef6cc2f. Latest consensus state: current_term: 1 leader_uuid: "1e468515022a4d91bd4ca7b4aef6cc2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e468515022a4d91bd4ca7b4aef6cc2f" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 43789 } } }
I20260926 16:36:22.488142 28088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e468515022a4d91bd4ca7b4aef6cc2f [sys.catalog]: This master's current role is: LEADER
I20260926 16:36:22.493805 28091 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:36:22.499632 28091 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:36:22.508569 28091 catalog_manager.cc:1347] Generated new cluster ID: 4af2213ab23b4a1999c7b055fbff3dd8
I20260926 16:36:22.508697 28091 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:36:22.524132 28091 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:36:22.525614 28091 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:36:22.532992 28091 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e468515022a4d91bd4ca7b4aef6cc2f: Generated new TSK 0
I20260926 16:36:22.533824 28091 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:36:22.542287 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 28009
I20260926 16:36:22.543481 28071 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:36:22.636523 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 28009
W20260926 16:36:22.653738 27994 connection.cc:535] client connection to 127.6.33.62:43789 recv error: Network error: failed to read from TLS socket (remote: 127.6.33.62:43789): Connection reset by peer (error 104)
2026-09-26T16:36:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2026 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440233850748-6276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:36:22 dist-test-slave-khd3 krb5kdc[28108](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:36:22 dist-test-slave-khd3 krb5kdc[28108](info): set up 2 sockets
Sep 26 16:36:22 dist-test-slave-khd3 krb5kdc[28108](info): commencing operation
krb5kdc: starting...
W20260926 16:36:23.750133 6276 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Sep 26 16:36:23 dist-test-slave-khd3 krb5kdc[28108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:36:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:36:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab.
Entry for principal kudu/127.6.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab.
I20260926 16:36:24.029249 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440233850748-6276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440233850748-6276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440233850748-6276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:42787
--webserver_interface=127.6.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:35073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.33.62:42787
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:36:24.371626 28121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:36:24.372022 28121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:36:24.372195 28121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:36:24.379645 28121 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:36:24.379828 28121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:36:24.379958 28121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:36:24.380070 28121 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:36:24.380167 28121 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:36:24.390377 28121 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:35073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440233850748-6276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440233850748-6276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.33.62:42787
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:42787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440233850748-6276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:36:24.391844 28121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:36:24.393272 28121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:36:24.406111 28126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:36:24.405961 28127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:36:24.406086 28129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:36:24.406481 28121 server_base.cc:1034] running on GCE node
I20260926 16:36:24.407486 28121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:36:24.408989 28121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:36:24.410233 28121 hybrid_clock.cc:648] HybridClock initialized: now 1790440584410176 us; error 44 us; skew 500 ppm
I20260926 16:36:24.410607 28121 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:36:24 dist-test-slave-khd3 krb5kdc[28108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440584, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:36:24.414253 28121 init.cc:377] Logged in from keytab as kudu/127.6.33.62@KRBTEST.COM (short username kudu)
I20260926 16:36:24.415866 28121 webserver.cc:460] Webserver started at http://127.6.33.62:46317/ using document root <none> and password file <none>
I20260926 16:36:24.416623 28121 fs_manager.cc:359] Metadata directory not provided
I20260926 16:36:24.416826 28121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:36:24.417166 28121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:36:24.420333 28121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440233850748-6276-0/minicluster-data/master-0/data/instance:
uuid: "03c2fb89d52d4a51bf9830aa1e349137"
format_stamp: "Formatted at 2026-09-26 16:36:24 on dist-test-slave-khd3"
server_key: "155421f019a464ad1987d596680c15a0"
server_key_iv: "952f280e7f705d7424c7932e871f7492"
server_key_version: "encryptionkey@0"
I20260926 16:36:24.421254 28121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440233850748-6276-0/minicluster-data/master-0/wal/instance:
uuid: "03c2fb89d52d4a51bf9830aa1e349137"
format_stamp: "Formatted at 2026-09-26 16:36:24 on dist-test-slave-khd3"
server_key: "155421f019a464ad1987d596680c15a0"
server_key_iv: "952f280e7f705d7424c7932e871f7492"
server_key_version: "encryptionkey@0"
I20260926 16:36:24.426797 28121 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 16:36:24.430510 28136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:36:24.431902 28121 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 16:36:24.432089 28121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440233850748-6276-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440233850748-6276-0/minicluster-data/master-0/wal
uuid: "03c2fb89d52d4a51bf9830aa1e349137"
format_stamp: "Formatted at 2026-09-26 16:36:24 on dist-test-slave-khd3"
server_key: "155421f019a464ad1987d596680c15a0"
server_key_iv: "952f280e7f705d7424c7932e871f7492"
server_key_version: "encryptionkey@0"
I20260926 16:36:24.432294 28121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440233850748-6276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440233850748-6276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440233850748-6276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:36:24.454375 28121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:36:24.459033 28121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:36:24.459352 28121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:36:24.479012 28121 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.62:42787
I20260926 16:36:24.479024 28187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.62:42787 every 8 connection(s)
I20260926 16:36:24.480504 28121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
I20260926 16:36:24.484472 28188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:36:24.485056 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 28121
I20260926 16:36:24.485265 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440233850748-6276-0/minicluster-data/master-0/wal/instance
I20260926 16:36:24.485742 6276 external_mini_cluster.cc:1428] Setting key 3f7e0bda338e4e8733adffbc42263f8a
I20260926 16:36:24.498312 28188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03c2fb89d52d4a51bf9830aa1e349137: Bootstrap starting.
I20260926 16:36:24.499892 28191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:24.487709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54077 (local address 127.6.33.62:42787)
0926 16:36:24.488494 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:24.488510 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:24.488550 (+ 40us) server_negotiation.cc:407] Connection header received
0926 16:36:24.490521 (+ 1971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:24.490547 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:24.491006 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:24.491353 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:24.492023 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:24.493365 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:24.495279 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:24.496054 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:24.496191 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:24.497586 (+ 1395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:24.497593 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:24.497607 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:36:24.498148 (+ 541us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:24.498223 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:36:24.498247 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:24.498379 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:24.498734 (+ 355us) server_negotiation.cc:299] Negotiation successful
0926 16:36:24.498934 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":366,"threads_started":1}
I20260926 16:36:24.501653 28188 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03c2fb89d52d4a51bf9830aa1e349137: Neither blocks nor log segments found. Creating new log.
I20260926 16:36:24.502825 28188 log.cc:824] T 00000000000000000000000000000000 P 03c2fb89d52d4a51bf9830aa1e349137: Log is configured to *not* fsync() on all Append() calls
I20260926 16:36:24.503849 6276 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:36:24.505790 28188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03c2fb89d52d4a51bf9830aa1e349137: No bootstrap required, opened a new log
I20260926 16:36:24.515703 28191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:24.508350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47497 (local address 127.6.33.62:42787)
0926 16:36:24.508559 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:24.508571 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:24.509089 (+ 518us) server_negotiation.cc:407] Connection header received
0926 16:36:24.509252 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:24.509256 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:24.509315 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:24.509404 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:24.510085 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:24.510901 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:24.512895 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:24.513357 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:24.513457 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:24.514777 (+ 1320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:24.514782 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:24.514785 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:36:24.514814 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:24.514868 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:36:24.514890 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:24.515057 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:24.515243 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 16:36:24.515340 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 16:36:24.515682 28188 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03c2fb89d52d4a51bf9830aa1e349137 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03c2fb89d52d4a51bf9830aa1e349137" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 42787 } }
I20260926 16:36:24.516213 28188 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03c2fb89d52d4a51bf9830aa1e349137 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:36:24.516333 28188 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03c2fb89d52d4a51bf9830aa1e349137 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03c2fb89d52d4a51bf9830aa1e349137, State: Initialized, Role: FOLLOWER
I20260926 16:36:24.517123 28188 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03c2fb89d52d4a51bf9830aa1e349137 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03c2fb89d52d4a51bf9830aa1e349137" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 42787 } }
I20260926 16:36:24.517458 28188 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03c2fb89d52d4a51bf9830aa1e349137 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:36:24.517611 28188 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03c2fb89d52d4a51bf9830aa1e349137 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:36:24.517786 28188 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03c2fb89d52d4a51bf9830aa1e349137 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:36:24.520466 28188 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03c2fb89d52d4a51bf9830aa1e349137 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03c2fb89d52d4a51bf9830aa1e349137" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 42787 } }
I20260926 16:36:24.521019 28188 leader_election.cc:302] T 00000000000000000000000000000000 P 03c2fb89d52d4a51bf9830aa1e349137 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03c2fb89d52d4a51bf9830aa1e349137; no voters:
I20260926 16:36:24.521580 28188 leader_election.cc:288] T 00000000000000000000000000000000 P 03c2fb89d52d4a51bf9830aa1e349137 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:36:24.521736 28198 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03c2fb89d52d4a51bf9830aa1e349137 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:36:24.522469 28198 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03c2fb89d52d4a51bf9830aa1e349137 [term 1 LEADER]: Becoming Leader. State: Replica: 03c2fb89d52d4a51bf9830aa1e349137, State: Running, Role: LEADER
I20260926 16:36:24.523046 28198 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03c2fb89d52d4a51bf9830aa1e349137 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03c2fb89d52d4a51bf9830aa1e349137" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 42787 } }
I20260926 16:36:24.523749 28188 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03c2fb89d52d4a51bf9830aa1e349137 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:36:24.526010 28199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03c2fb89d52d4a51bf9830aa1e349137 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "03c2fb89d52d4a51bf9830aa1e349137" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03c2fb89d52d4a51bf9830aa1e349137" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 42787 } } }
I20260926 16:36:24.526510 28199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03c2fb89d52d4a51bf9830aa1e349137 [sys.catalog]: This master's current role is: LEADER
I20260926 16:36:24.526016 28200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03c2fb89d52d4a51bf9830aa1e349137 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03c2fb89d52d4a51bf9830aa1e349137. Latest consensus state: current_term: 1 leader_uuid: "03c2fb89d52d4a51bf9830aa1e349137" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03c2fb89d52d4a51bf9830aa1e349137" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 42787 } } }
I20260926 16:36:24.532521 28200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03c2fb89d52d4a51bf9830aa1e349137 [sys.catalog]: This master's current role is: LEADER
I20260926 16:36:24.533495 28204 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:36:24.541123 28204 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:36:24.549909 28204 catalog_manager.cc:1347] Generated new cluster ID: 3e7e8b7c213147a2aefcd5e6ee72ac86
I20260926 16:36:24.550055 28204 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:36:24.572137 28204 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:36:24.574107 28204 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:36:24.585307 28204 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03c2fb89d52d4a51bf9830aa1e349137: Generated new TSK 0
I20260926 16:36:24.586278 28204 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:36:24.600477 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 28121
W20260926 16:36:24.601866 28183 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54077
W20260926 16:36:24.602676 6276 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 28121: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 16:36:24.602802 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 28121
W20260926 16:36:24.620252 28106 connection.cc:535] client connection to 127.6.33.62:42787 recv error: Network error: failed to read from TLS socket (remote: 127.6.33.62:42787): Connection reset by peer (error 104)
2026-09-26T16:36:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1966 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440233850748-6276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:36:24 dist-test-slave-khd3 krb5kdc[28219](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:36:24 dist-test-slave-khd3 krb5kdc[28219](info): set up 2 sockets
Sep 26 16:36:24 dist-test-slave-khd3 krb5kdc[28219](info): commencing operation
krb5kdc: starting...
W20260926 16:36:25.730213 6276 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Sep 26 16:36:25 dist-test-slave-khd3 krb5kdc[28219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:36:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:36:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab.
Entry for principal kudu/127.6.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab.
I20260926 16:36:25.992592 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440233850748-6276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440233850748-6276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440233850748-6276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:40361
--webserver_interface=127.6.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:45301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.33.62:40361
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:36:26.328910 28232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:36:26.329285 28232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:36:26.329460 28232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:36:26.336874 28232 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:36:26.337055 28232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:36:26.337167 28232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:36:26.337282 28232 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:36:26.337370 28232 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:36:26.347391 28232 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:45301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440233850748-6276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440233850748-6276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.33.62:40361
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:40361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440233850748-6276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:36:26.348850 28232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:36:26.350211 28232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:36:26.362187 28237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:36:26.362262 28240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:36:26.362231 28238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:36:26.363574 28232 server_base.cc:1034] running on GCE node
I20260926 16:36:26.364404 28232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:36:26.366032 28232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:36:26.367250 28232 hybrid_clock.cc:648] HybridClock initialized: now 1790440586367198 us; error 40 us; skew 500 ppm
I20260926 16:36:26.367640 28232 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:36:26 dist-test-slave-khd3 krb5kdc[28219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440586, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:36:26.370831 28232 init.cc:377] Logged in from keytab as kudu/127.6.33.62@KRBTEST.COM (short username kudu)
I20260926 16:36:26.372413 28232 webserver.cc:460] Webserver started at http://127.6.33.62:42983/ using document root <none> and password file <none>
I20260926 16:36:26.373205 28232 fs_manager.cc:359] Metadata directory not provided
I20260926 16:36:26.373342 28232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:36:26.373660 28232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:36:26.376691 28232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440233850748-6276-0/minicluster-data/master-0/data/instance:
uuid: "38ece3e2d93d4208a7aeff2cdadc07a5"
format_stamp: "Formatted at 2026-09-26 16:36:26 on dist-test-slave-khd3"
server_key: "d8c6b8cf9159e81e0170b9f8ced6c402"
server_key_iv: "1b58e4c530254a0adafeac6c83fb2875"
server_key_version: "encryptionkey@0"
I20260926 16:36:26.377556 28232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440233850748-6276-0/minicluster-data/master-0/wal/instance:
uuid: "38ece3e2d93d4208a7aeff2cdadc07a5"
format_stamp: "Formatted at 2026-09-26 16:36:26 on dist-test-slave-khd3"
server_key: "d8c6b8cf9159e81e0170b9f8ced6c402"
server_key_iv: "1b58e4c530254a0adafeac6c83fb2875"
server_key_version: "encryptionkey@0"
I20260926 16:36:26.382752 28232 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 16:36:26.386518 28247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:36:26.387863 28232 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 16:36:26.388055 28232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440233850748-6276-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440233850748-6276-0/minicluster-data/master-0/wal
uuid: "38ece3e2d93d4208a7aeff2cdadc07a5"
format_stamp: "Formatted at 2026-09-26 16:36:26 on dist-test-slave-khd3"
server_key: "d8c6b8cf9159e81e0170b9f8ced6c402"
server_key_iv: "1b58e4c530254a0adafeac6c83fb2875"
server_key_version: "encryptionkey@0"
I20260926 16:36:26.388271 28232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440233850748-6276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440233850748-6276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440233850748-6276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:36:26.394182 28232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:36:26.398694 28232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:36:26.398993 28232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:36:26.417841 28232 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.62:40361
I20260926 16:36:26.417928 28298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.62:40361 every 8 connection(s)
I20260926 16:36:26.419294 28232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
I20260926 16:36:26.423035 28299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:36:26.427803 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 28232
I20260926 16:36:26.427999 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440233850748-6276-0/minicluster-data/master-0/wal/instance
I20260926 16:36:26.428551 6276 external_mini_cluster.cc:1428] Setting key f2ec92e5bb73c2342b5a93d2e4fcee28
I20260926 16:36:26.434695 28299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38ece3e2d93d4208a7aeff2cdadc07a5: Bootstrap starting.
I20260926 16:36:26.436517 28302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:26.430406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55695 (local address 127.6.33.62:40361)
0926 16:36:26.431207 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:26.431218 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:26.431261 (+ 43us) server_negotiation.cc:407] Connection header received
0926 16:36:26.433156 (+ 1895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:26.433180 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:26.433536 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:26.433855 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:26.434956 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:26.434962 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:26.434978 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:36:26.434993 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:26.435058 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:36:26.435078 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:26.435254 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:26.435354 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 16:36:26.435602 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":363,"threads_started":1}
I20260926 16:36:26.438107 28299 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38ece3e2d93d4208a7aeff2cdadc07a5: Neither blocks nor log segments found. Creating new log.
I20260926 16:36:26.439270 28299 log.cc:824] T 00000000000000000000000000000000 P 38ece3e2d93d4208a7aeff2cdadc07a5: Log is configured to *not* fsync() on all Append() calls
I20260926 16:36:26.440142 6276 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:36:26.442229 28299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38ece3e2d93d4208a7aeff2cdadc07a5: No bootstrap required, opened a new log
I20260926 16:36:26.447175 28302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:26.444387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33415 (local address 127.6.33.62:40361)
0926 16:36:26.444547 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:26.444551 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:26.445080 (+ 529us) server_negotiation.cc:407] Connection header received
0926 16:36:26.445272 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:26.445286 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:26.445337 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:26.445453 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:26.446451 (+ 998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:26.446457 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:26.446461 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:36:26.446485 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:26.446525 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:36:26.446544 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:26.446667 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:26.446781 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 16:36:26.446848 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 16:36:26.451211 28299 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38ece3e2d93d4208a7aeff2cdadc07a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38ece3e2d93d4208a7aeff2cdadc07a5" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 40361 } }
I20260926 16:36:26.451644 28299 raft_consensus.cc:374] T 00000000000000000000000000000000 P 38ece3e2d93d4208a7aeff2cdadc07a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:36:26.451761 28299 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38ece3e2d93d4208a7aeff2cdadc07a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38ece3e2d93d4208a7aeff2cdadc07a5, State: Initialized, Role: FOLLOWER
I20260926 16:36:26.452478 28299 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38ece3e2d93d4208a7aeff2cdadc07a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38ece3e2d93d4208a7aeff2cdadc07a5" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 40361 } }
I20260926 16:36:26.452795 28299 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38ece3e2d93d4208a7aeff2cdadc07a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:36:26.452941 28299 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38ece3e2d93d4208a7aeff2cdadc07a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:36:26.453135 28299 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38ece3e2d93d4208a7aeff2cdadc07a5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:36:26.455323 28299 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38ece3e2d93d4208a7aeff2cdadc07a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38ece3e2d93d4208a7aeff2cdadc07a5" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 40361 } }
I20260926 16:36:26.455945 28299 leader_election.cc:302] T 00000000000000000000000000000000 P 38ece3e2d93d4208a7aeff2cdadc07a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38ece3e2d93d4208a7aeff2cdadc07a5; no voters:
I20260926 16:36:26.456547 28299 leader_election.cc:288] T 00000000000000000000000000000000 P 38ece3e2d93d4208a7aeff2cdadc07a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:36:26.456691 28309 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38ece3e2d93d4208a7aeff2cdadc07a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:36:26.457468 28309 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38ece3e2d93d4208a7aeff2cdadc07a5 [term 1 LEADER]: Becoming Leader. State: Replica: 38ece3e2d93d4208a7aeff2cdadc07a5, State: Running, Role: LEADER
I20260926 16:36:26.457964 28309 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38ece3e2d93d4208a7aeff2cdadc07a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38ece3e2d93d4208a7aeff2cdadc07a5" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 40361 } }
I20260926 16:36:26.458767 28299 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38ece3e2d93d4208a7aeff2cdadc07a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:36:26.460670 28311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38ece3e2d93d4208a7aeff2cdadc07a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38ece3e2d93d4208a7aeff2cdadc07a5. Latest consensus state: current_term: 1 leader_uuid: "38ece3e2d93d4208a7aeff2cdadc07a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38ece3e2d93d4208a7aeff2cdadc07a5" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 40361 } } }
I20260926 16:36:26.460661 28310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38ece3e2d93d4208a7aeff2cdadc07a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38ece3e2d93d4208a7aeff2cdadc07a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38ece3e2d93d4208a7aeff2cdadc07a5" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 40361 } } }
I20260926 16:36:26.461189 28311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38ece3e2d93d4208a7aeff2cdadc07a5 [sys.catalog]: This master's current role is: LEADER
I20260926 16:36:26.461189 28310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38ece3e2d93d4208a7aeff2cdadc07a5 [sys.catalog]: This master's current role is: LEADER
I20260926 16:36:26.463914 28314 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:36:26.469084 28314 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:36:26.478926 28314 catalog_manager.cc:1347] Generated new cluster ID: ec98b4cc71634fdc80620bc9c7a20b05
I20260926 16:36:26.479064 28314 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:36:26.502983 28314 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:36:26.503509 28314 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:36:26.512183 28314 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38ece3e2d93d4208a7aeff2cdadc07a5: Generated new TSK 0
I20260926 16:36:26.513182 28314 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:36:26.526472 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 28232
W20260926 16:36:26.527734 28294 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55695
W20260926 16:36:26.528422 6276 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 28232: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 16:36:26.528579 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 28232
2026-09-26T16:36:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1924 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440233850748-6276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:36:26 dist-test-slave-khd3 krb5kdc[28330](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:36:26 dist-test-slave-khd3 krb5kdc[28330](info): set up 2 sockets
Sep 26 16:36:26 dist-test-slave-khd3 krb5kdc[28330](info): commencing operation
krb5kdc: starting...
W20260926 16:36:27.654315 6276 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Sep 26 16:36:27 dist-test-slave-khd3 krb5kdc[28330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:36:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:36:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab.
Entry for principal kudu/127.6.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab.
I20260926 16:36:27.920693 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440233850748-6276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440233850748-6276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440233850748-6276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:42923
--webserver_interface=127.6.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:37905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.33.62:42923
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:36:28.255857 28343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:36:28.256222 28343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:36:28.256320 28343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:36:28.264111 28343 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:36:28.264245 28343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:36:28.264298 28343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:36:28.264353 28343 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:36:28.264398 28343 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:36:28.274842 28343 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:37905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440233850748-6276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440233850748-6276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.33.62:42923
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:42923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440233850748-6276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:36:28.276152 28343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:36:28.277619 28343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 16:36:28.290367 28343 server_base.cc:1034] running on GCE node
W20260926 16:36:28.290287 28351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:36:28.290294 28349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:36:28.290728 28348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:36:28.291623 28343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:36:28.293283 28343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:36:28.294529 28343 hybrid_clock.cc:648] HybridClock initialized: now 1790440588294462 us; error 54 us; skew 500 ppm
I20260926 16:36:28.294962 28343 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:36:28 dist-test-slave-khd3 krb5kdc[28330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440588, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:36:28.298467 28343 init.cc:377] Logged in from keytab as kudu/127.6.33.62@KRBTEST.COM (short username kudu)
I20260926 16:36:28.300138 28343 webserver.cc:460] Webserver started at http://127.6.33.62:46573/ using document root <none> and password file <none>
I20260926 16:36:28.300951 28343 fs_manager.cc:359] Metadata directory not provided
I20260926 16:36:28.301065 28343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:36:28.301425 28343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:36:28.304601 28343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440233850748-6276-0/minicluster-data/master-0/data/instance:
uuid: "1383d8356b274f6896153eac0e5203f3"
format_stamp: "Formatted at 2026-09-26 16:36:28 on dist-test-slave-khd3"
server_key: "2a97e58ffb83e67c0080e2002271a58d"
server_key_iv: "a6e5f6ef05d2c981661b9e8bc5b3fd2e"
server_key_version: "encryptionkey@0"
I20260926 16:36:28.305451 28343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440233850748-6276-0/minicluster-data/master-0/wal/instance:
uuid: "1383d8356b274f6896153eac0e5203f3"
format_stamp: "Formatted at 2026-09-26 16:36:28 on dist-test-slave-khd3"
server_key: "2a97e58ffb83e67c0080e2002271a58d"
server_key_iv: "a6e5f6ef05d2c981661b9e8bc5b3fd2e"
server_key_version: "encryptionkey@0"
I20260926 16:36:28.311010 28343 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 16:36:28.314879 28358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:36:28.316395 28343 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 16:36:28.316612 28343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440233850748-6276-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440233850748-6276-0/minicluster-data/master-0/wal
uuid: "1383d8356b274f6896153eac0e5203f3"
format_stamp: "Formatted at 2026-09-26 16:36:28 on dist-test-slave-khd3"
server_key: "2a97e58ffb83e67c0080e2002271a58d"
server_key_iv: "a6e5f6ef05d2c981661b9e8bc5b3fd2e"
server_key_version: "encryptionkey@0"
I20260926 16:36:28.316903 28343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440233850748-6276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440233850748-6276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440233850748-6276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:36:28.330673 28343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:36:28.335125 28343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:36:28.335475 28343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:36:28.354642 28343 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.62:42923
I20260926 16:36:28.354738 28409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.62:42923 every 8 connection(s)
I20260926 16:36:28.356161 28343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
I20260926 16:36:28.359951 28410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:36:28.365753 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 28343
I20260926 16:36:28.365940 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440233850748-6276-0/minicluster-data/master-0/wal/instance
I20260926 16:36:28.366510 6276 external_mini_cluster.cc:1428] Setting key 00bdcfa5d1a9cc562aaac82a085b8fa7
I20260926 16:36:28.371770 28410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1383d8356b274f6896153eac0e5203f3: Bootstrap starting.
I20260926 16:36:28.374927 28410 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1383d8356b274f6896153eac0e5203f3: Neither blocks nor log segments found. Creating new log.
I20260926 16:36:28.375946 28410 log.cc:824] T 00000000000000000000000000000000 P 1383d8356b274f6896153eac0e5203f3: Log is configured to *not* fsync() on all Append() calls
I20260926 16:36:28.378758 28410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1383d8356b274f6896153eac0e5203f3: No bootstrap required, opened a new log
Sep 26 16:36:28 dist-test-slave-khd3 krb5kdc[28330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:36:28.386346 28410 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1383d8356b274f6896153eac0e5203f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1383d8356b274f6896153eac0e5203f3" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 42923 } }
I20260926 16:36:28.386763 28410 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1383d8356b274f6896153eac0e5203f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:36:28.386889 28410 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1383d8356b274f6896153eac0e5203f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1383d8356b274f6896153eac0e5203f3, State: Initialized, Role: FOLLOWER
I20260926 16:36:28.387617 28410 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1383d8356b274f6896153eac0e5203f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1383d8356b274f6896153eac0e5203f3" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 42923 } }
I20260926 16:36:28.387884 28410 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1383d8356b274f6896153eac0e5203f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:36:28.388056 28410 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1383d8356b274f6896153eac0e5203f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:36:28.388268 28410 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1383d8356b274f6896153eac0e5203f3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:36:28.388809 28413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:28.368563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54689 (local address 127.6.33.62:42923)
0926 16:36:28.369529 (+ 966us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:28.369549 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:28.369597 (+ 48us) server_negotiation.cc:407] Connection header received
0926 16:36:28.371348 (+ 1751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:28.371378 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:28.371805 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:28.372281 (+ 476us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:28.373771 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:28.375098 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:28.376400 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:28.377189 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:28.377383 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:28.381375 (+ 3992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:28.381383 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:28.381397 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:36:28.381430 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:28.384611 (+ 3181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:28.385324 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:28.385334 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:28.385340 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:28.385504 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:28.385903 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:28.385913 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:28.385915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:28.386524 (+ 609us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:28.386831 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:28.387300 (+ 469us) server_negotiation.cc:299] Negotiation successful
0926 16:36:28.387718 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":736,"thread_start_us":469,"threads_started":1}
I20260926 16:36:28.390909 28410 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1383d8356b274f6896153eac0e5203f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1383d8356b274f6896153eac0e5203f3" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 42923 } }
I20260926 16:36:28.391469 28410 leader_election.cc:302] T 00000000000000000000000000000000 P 1383d8356b274f6896153eac0e5203f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1383d8356b274f6896153eac0e5203f3; no voters:
I20260926 16:36:28.392195 28410 leader_election.cc:288] T 00000000000000000000000000000000 P 1383d8356b274f6896153eac0e5203f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:36:28.392331 28415 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1383d8356b274f6896153eac0e5203f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:36:28.392634 6276 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:36:28.393493 28415 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1383d8356b274f6896153eac0e5203f3 [term 1 LEADER]: Becoming Leader. State: Replica: 1383d8356b274f6896153eac0e5203f3, State: Running, Role: LEADER
I20260926 16:36:28.394089 28415 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1383d8356b274f6896153eac0e5203f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1383d8356b274f6896153eac0e5203f3" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 42923 } }
I20260926 16:36:28.394678 28410 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1383d8356b274f6896153eac0e5203f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:36:28.401611 28417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1383d8356b274f6896153eac0e5203f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1383d8356b274f6896153eac0e5203f3. Latest consensus state: current_term: 1 leader_uuid: "1383d8356b274f6896153eac0e5203f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1383d8356b274f6896153eac0e5203f3" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 42923 } } }
I20260926 16:36:28.402055 28417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1383d8356b274f6896153eac0e5203f3 [sys.catalog]: This master's current role is: LEADER
I20260926 16:36:28.401005 28416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1383d8356b274f6896153eac0e5203f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1383d8356b274f6896153eac0e5203f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1383d8356b274f6896153eac0e5203f3" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 42923 } } }
I20260926 16:36:28.402652 28416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1383d8356b274f6896153eac0e5203f3 [sys.catalog]: This master's current role is: LEADER
I20260926 16:36:28.406505 28428 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:36:28.412256 28428 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:36:28.422196 28413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:28.401762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48835 (local address 127.6.33.62:42923)
0926 16:36:28.403564 (+ 1802us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:28.403574 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:28.404167 (+ 593us) server_negotiation.cc:407] Connection header received
0926 16:36:28.404385 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:28.404392 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:28.404498 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:28.404618 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:28.406954 (+ 2336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:28.408126 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:28.410069 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:28.411191 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:28.411389 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:28.414803 (+ 3414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:28.414811 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:28.414816 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:36:28.414861 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:28.418813 (+ 3952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:28.419713 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:28.419718 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:28.419724 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:28.419855 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:28.420256 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:28.420263 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:28.420271 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:28.420839 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:28.421076 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:28.421486 (+ 410us) server_negotiation.cc:299] Negotiation successful
0926 16:36:28.421823 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1604}
I20260926 16:36:28.423645 28428 catalog_manager.cc:1347] Generated new cluster ID: 55bc9962b244430382b3af0c347f2a30
I20260926 16:36:28.423764 28428 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:36:28.455523 28428 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:36:28.457394 28428 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:36:28.465603 28428 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1383d8356b274f6896153eac0e5203f3: Generated new TSK 0
I20260926 16:36:28.466480 28428 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:36:28.479193 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 28343
I20260926 16:36:28.480249 28405 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:36:28.573643 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 28343
2026-09-26T16:36:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2045 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440233850748-6276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:36:28 dist-test-slave-khd3 krb5kdc[28442](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:36:28 dist-test-slave-khd3 krb5kdc[28442](info): set up 2 sockets
Sep 26 16:36:28 dist-test-slave-khd3 krb5kdc[28442](info): commencing operation
krb5kdc: starting...
W20260926 16:36:29.690274 6276 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Sep 26 16:36:29 dist-test-slave-khd3 krb5kdc[28442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:36:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:36:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab.
Entry for principal kudu/127.6.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab.
I20260926 16:36:29.964445 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440233850748-6276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440233850748-6276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440233850748-6276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:39081
--webserver_interface=127.6.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:45405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.33.62:39081
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:36:30.303491 28455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:36:30.303879 28455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:36:30.304062 28455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:36:30.311697 28455 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:36:30.311882 28455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:36:30.312012 28455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:36:30.312120 28455 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:36:30.312208 28455 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:36:30.323123 28455 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:45405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440233850748-6276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440233850748-6276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.33.62:39081
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:39081
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440233850748-6276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:36:30.324517 28455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:36:30.325928 28455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:36:30.338258 28463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:36:30.338420 28460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:36:30.338420 28461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:36:30.339258 28455 server_base.cc:1034] running on GCE node
I20260926 16:36:30.340238 28455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:36:30.341897 28455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:36:30.343148 28455 hybrid_clock.cc:648] HybridClock initialized: now 1790440590343103 us; error 43 us; skew 500 ppm
I20260926 16:36:30.343533 28455 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:36:30 dist-test-slave-khd3 krb5kdc[28442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440590, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:36:30.363118 28455 init.cc:377] Logged in from keytab as kudu/127.6.33.62@KRBTEST.COM (short username kudu)
I20260926 16:36:30.364971 28455 webserver.cc:460] Webserver started at http://127.6.33.62:46793/ using document root <none> and password file <none>
I20260926 16:36:30.365741 28455 fs_manager.cc:359] Metadata directory not provided
I20260926 16:36:30.365873 28455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:36:30.366215 28455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:36:30.369421 28455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440233850748-6276-0/minicluster-data/master-0/data/instance:
uuid: "564464597de84b71b02827f1066aaf07"
format_stamp: "Formatted at 2026-09-26 16:36:30 on dist-test-slave-khd3"
server_key: "cade6e1bc52cf1e0ba4f96ec2f162d08"
server_key_iv: "1a63fe5db0af0e6b95848e16c20e6f54"
server_key_version: "encryptionkey@0"
I20260926 16:36:30.370203 28455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440233850748-6276-0/minicluster-data/master-0/wal/instance:
uuid: "564464597de84b71b02827f1066aaf07"
format_stamp: "Formatted at 2026-09-26 16:36:30 on dist-test-slave-khd3"
server_key: "cade6e1bc52cf1e0ba4f96ec2f162d08"
server_key_iv: "1a63fe5db0af0e6b95848e16c20e6f54"
server_key_version: "encryptionkey@0"
I20260926 16:36:30.375742 28455 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 16:36:30.379509 28470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:36:30.381131 28455 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:36:30.381322 28455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440233850748-6276-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440233850748-6276-0/minicluster-data/master-0/wal
uuid: "564464597de84b71b02827f1066aaf07"
format_stamp: "Formatted at 2026-09-26 16:36:30 on dist-test-slave-khd3"
server_key: "cade6e1bc52cf1e0ba4f96ec2f162d08"
server_key_iv: "1a63fe5db0af0e6b95848e16c20e6f54"
server_key_version: "encryptionkey@0"
I20260926 16:36:30.381525 28455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440233850748-6276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440233850748-6276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440233850748-6276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:36:30.398783 28455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:36:30.403362 28455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:36:30.403667 28455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:36:30.423440 28455 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.62:39081
I20260926 16:36:30.423470 28521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.62:39081 every 8 connection(s)
I20260926 16:36:30.424988 28455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
I20260926 16:36:30.428862 28522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:36:30.431396 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 28455
I20260926 16:36:30.431568 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440233850748-6276-0/minicluster-data/master-0/wal/instance
I20260926 16:36:30.432137 6276 external_mini_cluster.cc:1428] Setting key e0f44431ef06dbca9065bcc6053c0722
I20260926 16:36:30.441031 28522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 564464597de84b71b02827f1066aaf07: Bootstrap starting.
I20260926 16:36:30.444126 28522 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 564464597de84b71b02827f1066aaf07: Neither blocks nor log segments found. Creating new log.
I20260926 16:36:30.445314 28525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:30.434138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58767 (local address 127.6.33.62:39081)
0926 16:36:30.434941 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:30.434956 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:30.434999 (+ 43us) server_negotiation.cc:407] Connection header received
0926 16:36:30.436680 (+ 1681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:30.436703 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:30.437202 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:30.437589 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:30.438152 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:30.439471 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:30.440815 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:30.441586 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:30.441792 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:30.442979 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:30.442986 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:30.443001 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:36:30.443526 (+ 525us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:30.443601 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:36:30.443631 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:30.443754 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:30.443971 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 16:36:30.444276 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":337,"threads_started":1}
I20260926 16:36:30.445348 28522 log.cc:824] T 00000000000000000000000000000000 P 564464597de84b71b02827f1066aaf07: Log is configured to *not* fsync() on all Append() calls
I20260926 16:36:30.448285 28522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 564464597de84b71b02827f1066aaf07: No bootstrap required, opened a new log
I20260926 16:36:30.448941 6276 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:36:30.456631 28522 raft_consensus.cc:348] T 00000000000000000000000000000000 P 564464597de84b71b02827f1066aaf07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "564464597de84b71b02827f1066aaf07" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 39081 } }
I20260926 16:36:30.457211 28522 raft_consensus.cc:374] T 00000000000000000000000000000000 P 564464597de84b71b02827f1066aaf07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:36:30.457353 28522 raft_consensus.cc:729] T 00000000000000000000000000000000 P 564464597de84b71b02827f1066aaf07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 564464597de84b71b02827f1066aaf07, State: Initialized, Role: FOLLOWER
I20260926 16:36:30.457998 28522 consensus_queue.cc:260] T 00000000000000000000000000000000 P 564464597de84b71b02827f1066aaf07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "564464597de84b71b02827f1066aaf07" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 39081 } }
I20260926 16:36:30.458262 28522 raft_consensus.cc:388] T 00000000000000000000000000000000 P 564464597de84b71b02827f1066aaf07 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:36:30.458436 28522 raft_consensus.cc:482] T 00000000000000000000000000000000 P 564464597de84b71b02827f1066aaf07 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:36:30.458627 28522 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 564464597de84b71b02827f1066aaf07 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:36:30.459489 28525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:30.453148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60739 (local address 127.6.33.62:39081)
0926 16:36:30.453338 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:30.453346 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:30.453786 (+ 440us) server_negotiation.cc:407] Connection header received
0926 16:36:30.454009 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:30.454015 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:30.454106 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:30.454215 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:30.454713 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:30.455462 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:30.456753 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:30.457355 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:30.457477 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:30.458725 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:30.458734 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:30.458739 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:36:30.458762 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:30.458814 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:36:30.458836 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:30.458951 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:30.459029 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 16:36:30.459143 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 16:36:30.461625 28522 raft_consensus.cc:504] T 00000000000000000000000000000000 P 564464597de84b71b02827f1066aaf07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "564464597de84b71b02827f1066aaf07" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 39081 } }
I20260926 16:36:30.462141 28522 leader_election.cc:302] T 00000000000000000000000000000000 P 564464597de84b71b02827f1066aaf07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 564464597de84b71b02827f1066aaf07; no voters:
I20260926 16:36:30.462734 28522 leader_election.cc:288] T 00000000000000000000000000000000 P 564464597de84b71b02827f1066aaf07 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:36:30.462904 28532 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 564464597de84b71b02827f1066aaf07 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:36:30.463596 28532 raft_consensus.cc:686] T 00000000000000000000000000000000 P 564464597de84b71b02827f1066aaf07 [term 1 LEADER]: Becoming Leader. State: Replica: 564464597de84b71b02827f1066aaf07, State: Running, Role: LEADER
I20260926 16:36:30.464170 28532 consensus_queue.cc:237] T 00000000000000000000000000000000 P 564464597de84b71b02827f1066aaf07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "564464597de84b71b02827f1066aaf07" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 39081 } }
I20260926 16:36:30.464937 28522 sys_catalog.cc:564] T 00000000000000000000000000000000 P 564464597de84b71b02827f1066aaf07 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:36:30.466874 28534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 564464597de84b71b02827f1066aaf07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 564464597de84b71b02827f1066aaf07. Latest consensus state: current_term: 1 leader_uuid: "564464597de84b71b02827f1066aaf07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "564464597de84b71b02827f1066aaf07" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 39081 } } }
I20260926 16:36:30.467159 28534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 564464597de84b71b02827f1066aaf07 [sys.catalog]: This master's current role is: LEADER
I20260926 16:36:30.468178 28537 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:36:30.469619 28533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 564464597de84b71b02827f1066aaf07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "564464597de84b71b02827f1066aaf07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "564464597de84b71b02827f1066aaf07" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 39081 } } }
I20260926 16:36:30.469931 28533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 564464597de84b71b02827f1066aaf07 [sys.catalog]: This master's current role is: LEADER
I20260926 16:36:30.477864 28537 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:36:30.489377 28537 catalog_manager.cc:1347] Generated new cluster ID: fcdb27deb93c48ea8e18ab6424cfcc7b
I20260926 16:36:30.489569 28537 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:36:30.525414 28537 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:36:30.527442 28537 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:36:30.537310 28537 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 564464597de84b71b02827f1066aaf07: Generated new TSK 0
I20260926 16:36:30.538380 28537 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:36:30.555315 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 28455
W20260926 16:36:30.556613 28517 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:58767
W20260926 16:36:30.557456 6276 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 28455: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 16:36:30.557613 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 28455
2026-09-26T16:36:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1983 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440233850748-6276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:36:30 dist-test-slave-khd3 krb5kdc[28553](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:36:30 dist-test-slave-khd3 krb5kdc[28553](info): set up 2 sockets
Sep 26 16:36:30 dist-test-slave-khd3 krb5kdc[28553](info): commencing operation
krb5kdc: starting...
W20260926 16:36:31.678464 6276 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Sep 26 16:36:31 dist-test-slave-khd3 krb5kdc[28553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440591, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:36:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:36:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab.
Entry for principal kudu/127.6.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab.
I20260926 16:36:31.948936 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440233850748-6276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440233850748-6276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440233850748-6276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:34827
--webserver_interface=127.6.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:39067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.33.62:34827
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:36:32.286924 28566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:36:32.287237 28566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:36:32.287330 28566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:36:32.294935 28566 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:36:32.295106 28566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:36:32.295159 28566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:36:32.295225 28566 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:36:32.295274 28566 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:36:32.305233 28566 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:39067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440233850748-6276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440233850748-6276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.33.62:34827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:34827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440233850748-6276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:36:32.306493 28566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:36:32.307749 28566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:36:32.319758 28572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:36:32.319826 28571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:36:32.319758 28574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:36:32.320573 28566 server_base.cc:1034] running on GCE node
I20260926 16:36:32.321408 28566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:36:32.322791 28566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:36:32.324084 28566 hybrid_clock.cc:648] HybridClock initialized: now 1790440592324039 us; error 42 us; skew 500 ppm
I20260926 16:36:32.324564 28566 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:36:32 dist-test-slave-khd3 krb5kdc[28553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440592, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:36:32.327860 28566 init.cc:377] Logged in from keytab as kudu/127.6.33.62@KRBTEST.COM (short username kudu)
I20260926 16:36:32.329511 28566 webserver.cc:460] Webserver started at http://127.6.33.62:46463/ using document root <none> and password file <none>
I20260926 16:36:32.330210 28566 fs_manager.cc:359] Metadata directory not provided
I20260926 16:36:32.330355 28566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:36:32.330667 28566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:36:32.333773 28566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440233850748-6276-0/minicluster-data/master-0/data/instance:
uuid: "a6f7ed238caa4f81ba3d7849d3cebb5b"
format_stamp: "Formatted at 2026-09-26 16:36:32 on dist-test-slave-khd3"
server_key: "4e9925fc6f3c37510407180aa3711106"
server_key_iv: "6ae0efb7a8868a47ebcb77635e7335e1"
server_key_version: "encryptionkey@0"
I20260926 16:36:32.334559 28566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440233850748-6276-0/minicluster-data/master-0/wal/instance:
uuid: "a6f7ed238caa4f81ba3d7849d3cebb5b"
format_stamp: "Formatted at 2026-09-26 16:36:32 on dist-test-slave-khd3"
server_key: "4e9925fc6f3c37510407180aa3711106"
server_key_iv: "6ae0efb7a8868a47ebcb77635e7335e1"
server_key_version: "encryptionkey@0"
I20260926 16:36:32.339977 28566 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 16:36:32.343812 28581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:36:32.345207 28566 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:36:32.345391 28566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440233850748-6276-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440233850748-6276-0/minicluster-data/master-0/wal
uuid: "a6f7ed238caa4f81ba3d7849d3cebb5b"
format_stamp: "Formatted at 2026-09-26 16:36:32 on dist-test-slave-khd3"
server_key: "4e9925fc6f3c37510407180aa3711106"
server_key_iv: "6ae0efb7a8868a47ebcb77635e7335e1"
server_key_version: "encryptionkey@0"
I20260926 16:36:32.345599 28566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440233850748-6276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440233850748-6276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440233850748-6276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:36:32.351382 28566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:36:32.355907 28566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:36:32.356235 28566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:36:32.375094 28566 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.62:34827
I20260926 16:36:32.375150 28632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.62:34827 every 8 connection(s)
I20260926 16:36:32.376513 28566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
I20260926 16:36:32.380522 28633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:36:32.384379 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 28566
I20260926 16:36:32.384569 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440233850748-6276-0/minicluster-data/master-0/wal/instance
I20260926 16:36:32.385118 6276 external_mini_cluster.cc:1428] Setting key 64b30fd645161d7b2e2d3220895b3b2c
I20260926 16:36:32.393271 28636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:32.386980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35027 (local address 127.6.33.62:34827)
0926 16:36:32.387765 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:32.387777 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:32.387817 (+ 40us) server_negotiation.cc:407] Connection header received
0926 16:36:32.389532 (+ 1715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:32.389562 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:32.389929 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:32.390266 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:32.391369 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:32.391380 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:32.391399 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:36:32.391421 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:32.391510 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:36:32.391544 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:32.391790 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:32.391957 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 16:36:32.392196 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":347,"threads_started":1}
I20260926 16:36:32.394045 28633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6f7ed238caa4f81ba3d7849d3cebb5b: Bootstrap starting.
I20260926 16:36:32.396909 6276 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:36:32.397439 28633 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6f7ed238caa4f81ba3d7849d3cebb5b: Neither blocks nor log segments found. Creating new log.
I20260926 16:36:32.398453 28633 log.cc:824] T 00000000000000000000000000000000 P a6f7ed238caa4f81ba3d7849d3cebb5b: Log is configured to *not* fsync() on all Append() calls
I20260926 16:36:32.401859 28633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6f7ed238caa4f81ba3d7849d3cebb5b: No bootstrap required, opened a new log
I20260926 16:36:32.403894 28636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:32.401259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42659 (local address 127.6.33.62:34827)
0926 16:36:32.401444 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:32.401453 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:32.401833 (+ 380us) server_negotiation.cc:407] Connection header received
0926 16:36:32.402038 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:32.402043 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:32.402096 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:32.402199 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:32.403198 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:32.403204 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:32.403210 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:36:32.403233 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:32.403277 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:36:32.403293 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:32.403441 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:32.403543 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 16:36:32.403607 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 16:36:32.410300 28633 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6f7ed238caa4f81ba3d7849d3cebb5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6f7ed238caa4f81ba3d7849d3cebb5b" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 34827 } }
I20260926 16:36:32.410728 28633 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6f7ed238caa4f81ba3d7849d3cebb5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:36:32.410848 28633 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6f7ed238caa4f81ba3d7849d3cebb5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6f7ed238caa4f81ba3d7849d3cebb5b, State: Initialized, Role: FOLLOWER
I20260926 16:36:32.411554 28633 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6f7ed238caa4f81ba3d7849d3cebb5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6f7ed238caa4f81ba3d7849d3cebb5b" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 34827 } }
I20260926 16:36:32.411891 28633 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6f7ed238caa4f81ba3d7849d3cebb5b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:36:32.412068 28633 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6f7ed238caa4f81ba3d7849d3cebb5b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:36:32.412241 28633 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6f7ed238caa4f81ba3d7849d3cebb5b [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:36:32.414611 28633 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6f7ed238caa4f81ba3d7849d3cebb5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6f7ed238caa4f81ba3d7849d3cebb5b" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 34827 } }
I20260926 16:36:32.415158 28633 leader_election.cc:302] T 00000000000000000000000000000000 P a6f7ed238caa4f81ba3d7849d3cebb5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6f7ed238caa4f81ba3d7849d3cebb5b; no voters:
I20260926 16:36:32.415762 28633 leader_election.cc:288] T 00000000000000000000000000000000 P a6f7ed238caa4f81ba3d7849d3cebb5b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:36:32.415935 28643 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6f7ed238caa4f81ba3d7849d3cebb5b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:36:32.416698 28643 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6f7ed238caa4f81ba3d7849d3cebb5b [term 1 LEADER]: Becoming Leader. State: Replica: a6f7ed238caa4f81ba3d7849d3cebb5b, State: Running, Role: LEADER
I20260926 16:36:32.417339 28643 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6f7ed238caa4f81ba3d7849d3cebb5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6f7ed238caa4f81ba3d7849d3cebb5b" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 34827 } }
I20260926 16:36:32.417900 28633 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6f7ed238caa4f81ba3d7849d3cebb5b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:36:32.420288 28644 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6f7ed238caa4f81ba3d7849d3cebb5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6f7ed238caa4f81ba3d7849d3cebb5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6f7ed238caa4f81ba3d7849d3cebb5b" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 34827 } } }
I20260926 16:36:32.420683 28645 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6f7ed238caa4f81ba3d7849d3cebb5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6f7ed238caa4f81ba3d7849d3cebb5b. Latest consensus state: current_term: 1 leader_uuid: "a6f7ed238caa4f81ba3d7849d3cebb5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6f7ed238caa4f81ba3d7849d3cebb5b" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 34827 } } }
I20260926 16:36:32.421165 28644 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6f7ed238caa4f81ba3d7849d3cebb5b [sys.catalog]: This master's current role is: LEADER
I20260926 16:36:32.421236 28645 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6f7ed238caa4f81ba3d7849d3cebb5b [sys.catalog]: This master's current role is: LEADER
I20260926 16:36:32.423098 28650 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:36:32.428972 28650 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:36:32.438791 28650 catalog_manager.cc:1347] Generated new cluster ID: df6515922ed0469eb5a765b07a1c60c0
I20260926 16:36:32.438925 28650 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:36:32.457604 28650 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:36:32.458066 28650 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:36:32.465318 28650 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6f7ed238caa4f81ba3d7849d3cebb5b: Generated new TSK 0
I20260926 16:36:32.466521 28650 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:36:32.475600 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 28566
W20260926 16:36:32.476783 28628 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:35027
W20260926 16:36:32.477555 6276 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 28566: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 16:36:32.477702 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 28566
2026-09-26T16:36:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1917 ms)
[----------] 18 tests from AuthTokenIssuingTest (35543 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:36:32 dist-test-slave-khd3 krb5kdc[28664](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:36:32 dist-test-slave-khd3 krb5kdc[28664](info): set up 2 sockets
Sep 26 16:36:32 dist-test-slave-khd3 krb5kdc[28664](info): commencing operation
krb5kdc: starting...
W20260926 16:36:33.598385 6276 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Sep 26 16:36:33 dist-test-slave-khd3 krb5kdc[28664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:36:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:36:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab.
Entry for principal kudu/127.6.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab.
I20260926 16:36:33.889454 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:34579
--webserver_interface=127.6.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:46763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.33.62:34579,127.6.33.61:33249,127.6.33.60:40133
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:36:34.225998 28677 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 16:36:34.226362 28677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:36:34.226516 28677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:36:34.226622 28677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:36:34.234261 28677 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:36:34.234351 28677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:36:34.234445 28677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:36:34.234525 28677 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:36:34.234617 28677 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:36:34.244531 28677 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:46763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.33.62:34579,127.6.33.61:33249,127.6.33.60:40133
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:34579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:36:34.245904 28677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:36:34.247300 28677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:36:34.259089 28683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:36:34.259037 28682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:36:34.259474 28685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:36:34.260236 28677 server_base.cc:1034] running on GCE node
I20260926 16:36:34.261085 28677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:36:34.262493 28677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:36:34.263715 28677 hybrid_clock.cc:648] HybridClock initialized: now 1790440594263652 us; error 38 us; skew 500 ppm
I20260926 16:36:34.264124 28677 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:36:34 dist-test-slave-khd3 krb5kdc[28664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440594, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:36:34.267310 28677 init.cc:377] Logged in from keytab as kudu/127.6.33.62@KRBTEST.COM (short username kudu)
I20260926 16:36:34.268831 28677 webserver.cc:460] Webserver started at http://127.6.33.62:42791/ using document root <none> and password file <none>
I20260926 16:36:34.269511 28677 fs_manager.cc:359] Metadata directory not provided
I20260926 16:36:34.269620 28677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:36:34.269941 28677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:36:34.272953 28677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/minicluster-data/master-0/data/instance:
uuid: "592bb7ca183c44dab01abef89341ddf2"
format_stamp: "Formatted at 2026-09-26 16:36:34 on dist-test-slave-khd3"
server_key: "5ec7595dbcccea09f859c893e31e2671"
server_key_iv: "68dcc0ad0baac0066c678cfb242e637b"
server_key_version: "encryptionkey@0"
I20260926 16:36:34.273999 28677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/minicluster-data/master-0/wal/instance:
uuid: "592bb7ca183c44dab01abef89341ddf2"
format_stamp: "Formatted at 2026-09-26 16:36:34 on dist-test-slave-khd3"
server_key: "5ec7595dbcccea09f859c893e31e2671"
server_key_iv: "68dcc0ad0baac0066c678cfb242e637b"
server_key_version: "encryptionkey@0"
I20260926 16:36:34.279590 28677 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 16:36:34.283286 28692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:36:34.284723 28677 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 16:36:34.284961 28677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/minicluster-data/master-0/wal
uuid: "592bb7ca183c44dab01abef89341ddf2"
format_stamp: "Formatted at 2026-09-26 16:36:34 on dist-test-slave-khd3"
server_key: "5ec7595dbcccea09f859c893e31e2671"
server_key_iv: "68dcc0ad0baac0066c678cfb242e637b"
server_key_version: "encryptionkey@0"
I20260926 16:36:34.285168 28677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:36:34.308175 28677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:36:34.312649 28677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:36:34.313053 28677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:36:34.332880 28677 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.62:34579
I20260926 16:36:34.332855 28743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.62:34579 every 8 connection(s)
I20260926 16:36:34.334343 28677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
I20260926 16:36:34.335599 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 28677
I20260926 16:36:34.335835 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/minicluster-data/master-0/wal/instance
I20260926 16:36:34.336624 6276 external_mini_cluster.cc:1428] Setting key 74ed737796e6c023d273e2b9c9340c5b
I20260926 16:36:34.339546 28744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:36:34.349779 28744 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.33.62" port: 34579 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.6.33.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.61@KRBTEST.COM" created.
Sep 26 16:36:34 dist-test-slave-khd3 krb5kdc[28664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440594, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.62@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:36:34.375344 28748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:34.353306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:34579 (local address 127.0.0.1:33650)
0926 16:36:34.354452 (+ 1146us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:34.354492 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 16:36:34.356285 (+ 1793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:36:34.357273 (+ 988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:36:34.357280 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:36:34.358020 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:36:34.358378 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:36:34.358395 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:36:34.359948 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:36:34.359953 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:36:34.361118 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:36:34.361130 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:36:34.362046 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:36:34.362053 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:36:34.362223 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:34.362911 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:36:34.362937 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:36:34.367385 (+ 4448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:36:34.370979 (+ 3594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:36:34.370998 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:36:34.371011 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:36:34.371433 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:36:34.371864 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:36:34.371871 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:36:34.371875 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:36:34.372045 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:36:34.372851 (+ 806us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:36:34.372859 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:36:34.373114 (+ 255us) client_negotiation.cc:769] Sending connection context
0926 16:36:34.373432 (+ 318us) client_negotiation.cc:240] Negotiation successful
0926 16:36:34.374220 (+ 788us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":934,"mutex_wait_us":188,"thread_start_us":721,"threads_started":1}
I20260926 16:36:34.375393 28747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:34.353306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33650 (local address 127.6.33.62:34579)
0926 16:36:34.354182 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:34.354203 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:34.354641 (+ 438us) server_negotiation.cc:407] Connection header received
0926 16:36:34.356684 (+ 2043us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:34.356690 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:34.357030 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:34.357262 (+ 232us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:34.358589 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:34.359794 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:34.361254 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:34.361816 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:34.362013 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:34.367631 (+ 5618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:34.367638 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:34.367651 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:36:34.367685 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:34.370650 (+ 2965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:34.371581 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:34.371588 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:34.371590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:34.371757 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:34.372159 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:34.372166 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:34.372169 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:34.372688 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:34.372925 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:34.373595 (+ 670us) server_negotiation.cc:299] Negotiation successful
0926 16:36:34.373990 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":380,"threads_started":1}
I20260926 16:36:34.378459 28744 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.33.61" port: 33249 } has no permanent_uuid. Determining permanent_uuid...
I20260926 16:36:34.379700 28748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:34.379190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.61:33249 (local address 127.0.0.1:48052)
0926 16:36:34.379401 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:34.379534 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.61:33249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260926 16:36:34.380009 28695 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.33.61:33249: connect: Connection refused (error 111)
I20260926 16:36:34.381779 28748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:34.381304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.61:33249 (local address 127.0.0.1:48066)
0926 16:36:34.381498 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:34.381605 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.61:33249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260926 16:36:34.381987 28744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.33.61:33249: Network error: Client connection negotiation failed: client connection to 127.6.33.61:33249: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.61.keytab.
Entry for principal kudu/127.6.33.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.61.keytab.
I20260926 16:36:34.400655 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.61:33249
--webserver_interface=127.6.33.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.61.keytab
--principal=kudu/127.6.33.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:46763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.33.62:34579,127.6.33.61:33249,127.6.33.60:40133
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:36:34.431373 28744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.33.61" port: 33249 } attempt: 1
I20260926 16:36:34.432479 28748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:34.432043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.61:33249 (local address 127.0.0.1:48068)
0926 16:36:34.432238 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:34.432331 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.61:33249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260926 16:36:34.433467 28748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:34.433131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.61:33249 (local address 127.0.0.1:48070)
0926 16:36:34.433274 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:34.433347 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.61:33249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 16:36:34.433651 28744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.33.61:33249: Network error: Client connection negotiation failed: client connection to 127.6.33.61:33249: connect: Connection refused (error 111)
I20260926 16:36:34.501907 28744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.33.61" port: 33249 } attempt: 2
I20260926 16:36:34.503086 28748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:34.502584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.61:33249 (local address 127.0.0.1:48078)
0926 16:36:34.502845 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:34.502929 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.61:33249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260926 16:36:34.504113 28748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:34.503723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.61:33249 (local address 127.0.0.1:48092)
0926 16:36:34.503888 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:34.503989 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.61:33249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 16:36:34.504303 28744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.33.61:33249: Network error: Client connection negotiation failed: client connection to 127.6.33.61:33249: connect: Connection refused (error 111)
I20260926 16:36:34.595571 28744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.33.61" port: 33249 } attempt: 3
I20260926 16:36:34.596724 28748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:34.596264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.61:33249 (local address 127.0.0.1:48106)
0926 16:36:34.596490 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:34.596579 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.61:33249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260926 16:36:34.597780 28748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:34.597433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.61:33249 (local address 127.0.0.1:48108)
0926 16:36:34.597576 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:34.597649 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.61:33249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 16:36:34.597967 28744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.33.61:33249: Network error: Client connection negotiation failed: client connection to 127.6.33.61:33249: connect: Connection refused (error 111)
W20260926 16:36:34.733764 28751 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 16:36:34.734172 28751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:36:34.734297 28751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:36:34.734460 28751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260926 16:36:34.741175 28744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.33.61" port: 33249 } attempt: 4
I20260926 16:36:34.742614 28748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:34.741924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.61:33249 (local address 127.0.0.1:48120)
0926 16:36:34.742329 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:34.742425 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.61:33249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265}
I20260926 16:36:34.743893 28748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:34.743445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.61:33249 (local address 127.0.0.1:48134)
0926 16:36:34.743637 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:34.743746 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.61:33249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260926 16:36:34.744143 28744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.33.61:33249: Network error: Client connection negotiation failed: client connection to 127.6.33.61:33249: connect: Connection refused (error 111)
W20260926 16:36:34.745618 28751 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:36:34.745697 28751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:36:34.745754 28751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:36:34.745803 28751 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:36:34.745864 28751 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:36:34.756320 28751 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:46763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.33.62:34579,127.6.33.61:33249,127.6.33.60:40133
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.61.keytab
--principal=kudu/127.6.33.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.61:33249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.33.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:36:34.757723 28751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:36:34.758981 28751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:36:34.772413 28757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:36:34.772464 28756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:36:34.772343 28759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:36:34.777204 28751 server_base.cc:1034] running on GCE node
I20260926 16:36:34.778064 28751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:36:34.779605 28751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:36:34.780875 28751 hybrid_clock.cc:648] HybridClock initialized: now 1790440594780778 us; error 61 us; skew 500 ppm
I20260926 16:36:34.781215 28751 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:36:34 dist-test-slave-khd3 krb5kdc[28664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440594, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:36:34.784507 28751 init.cc:377] Logged in from keytab as kudu/127.6.33.61@KRBTEST.COM (short username kudu)
I20260926 16:36:34.786221 28751 webserver.cc:460] Webserver started at http://127.6.33.61:35289/ using document root <none> and password file <none>
I20260926 16:36:34.786926 28751 fs_manager.cc:359] Metadata directory not provided
I20260926 16:36:34.787015 28751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:36:34.787290 28751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:36:34.790457 28751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/minicluster-data/master-1/data/instance:
uuid: "d5e35546b8d6472cb69b6465c4e09e63"
format_stamp: "Formatted at 2026-09-26 16:36:34 on dist-test-slave-khd3"
server_key: "cb782f9b25d4ffb6631f8fc87ee16a6e"
server_key_iv: "c2dbd1762b7e4cfa2855f4cea5c9c722"
server_key_version: "encryptionkey@0"
I20260926 16:36:34.791234 28751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/minicluster-data/master-1/wal/instance:
uuid: "d5e35546b8d6472cb69b6465c4e09e63"
format_stamp: "Formatted at 2026-09-26 16:36:34 on dist-test-slave-khd3"
server_key: "cb782f9b25d4ffb6631f8fc87ee16a6e"
server_key_iv: "c2dbd1762b7e4cfa2855f4cea5c9c722"
server_key_version: "encryptionkey@0"
I20260926 16:36:34.796640 28751 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 16:36:34.800289 28766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:36:34.801796 28751 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:36:34.801990 28751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/minicluster-data/master-1/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/minicluster-data/master-1/wal
uuid: "d5e35546b8d6472cb69b6465c4e09e63"
format_stamp: "Formatted at 2026-09-26 16:36:34 on dist-test-slave-khd3"
server_key: "cb782f9b25d4ffb6631f8fc87ee16a6e"
server_key_iv: "c2dbd1762b7e4cfa2855f4cea5c9c722"
server_key_version: "encryptionkey@0"
I20260926 16:36:34.802265 28751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:36:34.823727 28751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:36:34.828240 28751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:36:34.828552 28751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:36:34.847251 28751 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.61:33249
I20260926 16:36:34.847273 28817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.61:33249 every 8 connection(s)
I20260926 16:36:34.848681 28751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/minicluster-data/master-1/data/info.pb
I20260926 16:36:34.852582 28818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:36:34.856729 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 28751
I20260926 16:36:34.856961 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/minicluster-data/master-1/wal/instance
I20260926 16:36:34.857522 6276 external_mini_cluster.cc:1428] Setting key e15205b10ffed59c4935a5e254cb4044
I20260926 16:36:34.861205 28818 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.33.62" port: 34579 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 16:36:34 dist-test-slave-khd3 krb5kdc[28664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440594, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.61@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.33.60@KRBTEST.COM; defaulting to no policy
I20260926 16:36:34.883879 28747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:34.864659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33660 (local address 127.6.33.62:34579)
0926 16:36:34.864910 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:34.864916 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:34.866317 (+ 1401us) server_negotiation.cc:407] Connection header received
0926 16:36:34.868662 (+ 2345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:34.868667 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:34.868738 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:34.868883 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:34.870548 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:34.871276 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:34.873030 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:34.873447 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:34.873585 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:34.877333 (+ 3748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:34.877339 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:34.877342 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:36:34.877380 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:34.880536 (+ 3156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:34.881398 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:34.881404 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:34.881407 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:34.881537 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:34.881993 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:34.881997 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:34.881999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:34.882336 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:34.882497 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:34.883307 (+ 810us) server_negotiation.cc:299] Negotiation successful
0926 16:36:34.883539 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.60@KRBTEST.COM" created.
I20260926 16:36:34.885196 28820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:34.865255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:34579 (local address 127.0.0.1:33660)
0926 16:36:34.866116 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:34.866165 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 16:36:34.868365 (+ 2200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:36:34.869114 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:36:34.869122 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:36:34.869746 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:36:34.870362 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:36:34.870381 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:36:34.871451 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:36:34.871455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:36:34.872876 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:36:34.872889 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:36:34.873688 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:36:34.873695 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:36:34.873934 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:34.874651 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:36:34.874679 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:36:34.877068 (+ 2389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:36:34.880706 (+ 3638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:36:34.880716 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:36:34.880735 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:36:34.881258 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:36:34.881654 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:36:34.881659 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:36:34.881663 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:36:34.881860 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:36:34.882472 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:36:34.882480 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:36:34.882876 (+ 396us) client_negotiation.cc:769] Sending connection context
0926 16:36:34.883207 (+ 331us) client_negotiation.cc:240] Negotiation successful
0926 16:36:34.883778 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":340,"threads_started":1}
I20260926 16:36:34.887167 28818 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.33.61" port: 33249 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 16:36:34 dist-test-slave-khd3 krb5kdc[28664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440594, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.61@KRBTEST.COM for kudu/127.6.33.61@KRBTEST.COM
I20260926 16:36:34.905846 28820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:34.888011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.61:33249 (local address 127.0.0.1:48140)
0926 16:36:34.888214 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:34.888237 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 16:36:34.888351 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:36:34.889725 (+ 1374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:36:34.889733 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:36:34.890285 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:36:34.890509 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:36:34.890525 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:36:34.891994 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:36:34.892003 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:36:34.893620 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:36:34.893639 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:36:34.894947 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:36:34.894951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:36:34.895167 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:34.895835 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:36:34.895854 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:36:34.898128 (+ 2274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:36:34.902953 (+ 4825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:36:34.902959 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:36:34.902962 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:36:34.903415 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:36:34.903926 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:36:34.903930 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:36:34.903933 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:36:34.904089 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:36:34.904970 (+ 881us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:36:34.904974 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:36:34.905130 (+ 156us) client_negotiation.cc:769] Sending connection context
0926 16:36:34.905254 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 16:36:34.905530 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72,"mutex_wait_us":41}
I20260926 16:36:34.906329 28822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:34.888324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48140 (local address 127.6.33.61:33249)
0926 16:36:34.889219 (+ 895us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:34.889226 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:34.889255 (+ 29us) server_negotiation.cc:407] Connection header received
0926 16:36:34.889303 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:34.889309 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:34.889534 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:34.889688 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:34.890727 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:34.891833 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:34.893786 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:34.894772 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:34.894921 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:34.898385 (+ 3464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:34.898406 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:34.898430 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:36:34.898481 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:34.902770 (+ 4289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:34.903595 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:34.903615 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:34.903621 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:34.903778 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:34.904223 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:34.904231 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:34.904238 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:34.904814 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:34.905055 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:34.905466 (+ 411us) server_negotiation.cc:299] Negotiation successful
0926 16:36:34.905779 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":738,"thread_start_us":569,"threads_started":1}
I20260926 16:36:34.909168 28818 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.33.60" port: 40133 } has no permanent_uuid. Determining permanent_uuid...
I20260926 16:36:34.910671 28820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:34.909993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:40133 (local address 127.0.0.1:45472)
0926 16:36:34.910261 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:34.910441 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.60:40133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
W20260926 16:36:34.910986 28768 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.33.60:40133: connect: Connection refused (error 111)
I20260926 16:36:34.912806 28820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:34.912297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:40133 (local address 127.0.0.1:45482)
0926 16:36:34.912508 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:34.912626 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.60:40133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260926 16:36:34.913094 28818 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.33.60:40133: Network error: Client connection negotiation failed: client connection to 127.6.33.60:40133: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.60.keytab.
Entry for principal kudu/127.6.33.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.60.keytab.
I20260926 16:36:34.923592 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.60:40133
--webserver_interface=127.6.33.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.60.keytab
--principal=kudu/127.6.33.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:46763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.33.62:34579,127.6.33.61:33249,127.6.33.60:40133
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:36:34.962461 28818 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.33.60" port: 40133 } attempt: 1
I20260926 16:36:34.963743 28820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:34.963202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:40133 (local address 127.0.0.1:45484)
0926 16:36:34.963431 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:34.963538 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.60:40133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260926 16:36:34.964834 28820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:34.964393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:40133 (local address 127.0.0.1:45486)
0926 16:36:34.964562 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:34.964660 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.60:40133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 16:36:34.965160 28818 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.33.60:40133: Network error: Client connection negotiation failed: client connection to 127.6.33.60:40133: connect: Connection refused (error 111)
I20260926 16:36:35.033407 28818 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.33.60" port: 40133 } attempt: 2
I20260926 16:36:35.034842 28820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:35.034278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:40133 (local address 127.0.0.1:45500)
0926 16:36:35.034521 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:35.034650 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.60:40133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260926 16:36:35.036007 28820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:35.035603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:40133 (local address 127.0.0.1:45502)
0926 16:36:35.035771 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:35.035861 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.60:40133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260926 16:36:35.036300 28818 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.33.60:40133: Network error: Client connection negotiation failed: client connection to 127.6.33.60:40133: connect: Connection refused (error 111)
I20260926 16:36:35.043406 28744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.33.61" port: 33249 } attempt: 5
Sep 26 16:36:35 dist-test-slave-khd3 krb5kdc[28664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440594, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.62@KRBTEST.COM for kudu/127.6.33.61@KRBTEST.COM
I20260926 16:36:35.058508 28822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:35.044377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48142 (local address 127.6.33.61:33249)
0926 16:36:35.044549 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:35.044553 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:35.044572 (+ 19us) server_negotiation.cc:407] Connection header received
0926 16:36:35.044639 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:35.044643 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:35.044701 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:35.044858 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:35.045798 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:35.046509 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:35.048195 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:35.048678 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:35.048892 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:35.052561 (+ 3669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:35.052567 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:35.052570 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:36:35.052607 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:35.055697 (+ 3090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:35.056412 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:35.056417 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:35.056421 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:35.056552 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:35.057018 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:35.057025 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:35.057028 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:35.057358 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:35.057548 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:35.057944 (+ 396us) server_negotiation.cc:299] Negotiation successful
0926 16:36:35.058162 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 16:36:35.058508 28748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:35.044138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.61:33249 (local address 127.0.0.1:48142)
0926 16:36:35.044355 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:35.044377 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 16:36:35.044511 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:36:35.044885 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:36:35.044890 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:36:35.045403 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:36:35.045636 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:36:35.045650 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:36:35.046680 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:36:35.046685 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:36:35.047995 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:36:35.048010 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:36:35.048875 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:36:35.048880 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:36:35.049101 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:35.049795 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:36:35.049818 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:36:35.052296 (+ 2478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:36:35.055860 (+ 3564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:36:35.055866 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:36:35.055871 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:36:35.056282 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:36:35.056671 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:36:35.056676 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:36:35.056680 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:36:35.056892 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:36:35.057524 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:36:35.057530 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:36:35.057695 (+ 165us) client_negotiation.cc:769] Sending connection context
0926 16:36:35.057840 (+ 145us) client_negotiation.cc:240] Negotiation successful
0926 16:36:35.058152 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260926 16:36:35.059681 28744 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.33.60" port: 40133 } has no permanent_uuid. Determining permanent_uuid...
I20260926 16:36:35.060868 28748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:35.060398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:40133 (local address 127.0.0.1:45512)
0926 16:36:35.060571 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:35.060658 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.60:40133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 16:36:35.061789 28748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:35.061456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:40133 (local address 127.0.0.1:45518)
0926 16:36:35.061595 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:35.061676 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.60:40133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 16:36:35.061995 28744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.33.60:40133: Network error: Client connection negotiation failed: client connection to 127.6.33.60:40133: connect: Connection refused (error 111)
I20260926 16:36:35.113317 28744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.33.60" port: 40133 } attempt: 1
I20260926 16:36:35.114516 28748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:35.114025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:40133 (local address 127.0.0.1:45520)
0926 16:36:35.114238 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:35.114335 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.60:40133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260926 16:36:35.115667 28748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:35.115305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:40133 (local address 127.0.0.1:45528)
0926 16:36:35.115471 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:35.115559 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.60:40133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260926 16:36:35.116043 28744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.33.60:40133: Network error: Client connection negotiation failed: client connection to 127.6.33.60:40133: connect: Connection refused (error 111)
I20260926 16:36:35.127569 28818 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.33.60" port: 40133 } attempt: 3
I20260926 16:36:35.128995 28820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:35.128290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:40133 (local address 127.0.0.1:45544)
0926 16:36:35.128600 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:35.128734 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.60:40133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260926 16:36:35.130291 28820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:35.129798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:40133 (local address 127.0.0.1:45560)
0926 16:36:35.130047 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:35.130146 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.60:40133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
W20260926 16:36:35.130549 28818 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.33.60:40133: Network error: Client connection negotiation failed: client connection to 127.6.33.60:40133: connect: Connection refused (error 111)
I20260926 16:36:35.184288 28744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.33.60" port: 40133 } attempt: 2
I20260926 16:36:35.185644 28748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:35.185127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:40133 (local address 127.0.0.1:45570)
0926 16:36:35.185350 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:35.185454 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.60:40133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260926 16:36:35.186780 28748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:35.186420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:40133 (local address 127.0.0.1:45584)
0926 16:36:35.186573 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:35.186665 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.60:40133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 16:36:35.187047 28744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.33.60:40133: Network error: Client connection negotiation failed: client connection to 127.6.33.60:40133: connect: Connection refused (error 111)
I20260926 16:36:35.273800 28818 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.33.60" port: 40133 } attempt: 4
W20260926 16:36:35.273830 28825 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 16:36:35.274295 28825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:36:35.274411 28825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:36:35.274570 28825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260926 16:36:35.275219 28820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:35.274661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:40133 (local address 127.0.0.1:45598)
0926 16:36:35.274916 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:35.275025 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.60:40133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260926 16:36:35.276309 28820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:35.275939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:40133 (local address 127.0.0.1:45610)
0926 16:36:35.276098 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:35.276182 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.60:40133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 16:36:35.276531 28818 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.33.60:40133: Network error: Client connection negotiation failed: client connection to 127.6.33.60:40133: connect: Connection refused (error 111)
W20260926 16:36:35.282909 28825 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:36:35.283015 28825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:36:35.283070 28825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:36:35.283114 28825 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:36:35.283169 28825 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:36:35.293268 28744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.33.60" port: 40133 } attempt: 3
I20260926 16:36:35.293440 28825 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:46763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.33.62:34579,127.6.33.61:33249,127.6.33.60:40133
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.60.keytab
--principal=kudu/127.6.33.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.60:40133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.33.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:36:35.294500 28748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:35.293991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:40133 (local address 127.0.0.1:45618)
0926 16:36:35.294238 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:35.294338 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.60:40133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260926 16:36:35.295044 28825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:36:35.295645 28748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:35.295135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:40133 (local address 127.0.0.1:45628)
0926 16:36:35.295391 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:35.295501 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.60:40133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
W20260926 16:36:35.295905 28744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.33.60:40133: Network error: Client connection negotiation failed: client connection to 127.6.33.60:40133: connect: Connection refused (error 111)
I20260926 16:36:35.296691 28825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:36:35.310714 28833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:36:35.310745 28831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:36:35.310846 28830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:36:35.311558 28825 server_base.cc:1034] running on GCE node
I20260926 16:36:35.312371 28825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:36:35.313939 28825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:36:35.315212 28825 hybrid_clock.cc:648] HybridClock initialized: now 1790440595315156 us; error 45 us; skew 500 ppm
I20260926 16:36:35.315579 28825 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:36:35 dist-test-slave-khd3 krb5kdc[28664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440595, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:36:35.319303 28825 init.cc:377] Logged in from keytab as kudu/127.6.33.60@KRBTEST.COM (short username kudu)
I20260926 16:36:35.320971 28825 webserver.cc:460] Webserver started at http://127.6.33.60:38469/ using document root <none> and password file <none>
I20260926 16:36:35.321700 28825 fs_manager.cc:359] Metadata directory not provided
I20260926 16:36:35.321832 28825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:36:35.322165 28825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:36:35.325250 28825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/minicluster-data/master-2/data/instance:
uuid: "c81bd5efa06746d382bb22825ef84fa4"
format_stamp: "Formatted at 2026-09-26 16:36:35 on dist-test-slave-khd3"
server_key: "de248d285c14e74ebbb5bf5d8953df8b"
server_key_iv: "6321a73375bf37dd1af4827d96053651"
server_key_version: "encryptionkey@0"
I20260926 16:36:35.326051 28825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/minicluster-data/master-2/wal/instance:
uuid: "c81bd5efa06746d382bb22825ef84fa4"
format_stamp: "Formatted at 2026-09-26 16:36:35 on dist-test-slave-khd3"
server_key: "de248d285c14e74ebbb5bf5d8953df8b"
server_key_iv: "6321a73375bf37dd1af4827d96053651"
server_key_version: "encryptionkey@0"
I20260926 16:36:35.331656 28825 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 16:36:35.335675 28840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:36:35.337368 28825 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 16:36:35.337561 28825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/minicluster-data/master-2/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/minicluster-data/master-2/wal
uuid: "c81bd5efa06746d382bb22825ef84fa4"
format_stamp: "Formatted at 2026-09-26 16:36:35 on dist-test-slave-khd3"
server_key: "de248d285c14e74ebbb5bf5d8953df8b"
server_key_iv: "6321a73375bf37dd1af4827d96053651"
server_key_version: "encryptionkey@0"
I20260926 16:36:35.337775 28825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:36:35.358839 28825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:36:35.370487 28825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:36:35.370805 28825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:36:35.390040 28891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.60:40133 every 8 connection(s)
I20260926 16:36:35.390046 28825 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.60:40133
I20260926 16:36:35.391628 28825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/minicluster-data/master-2/data/info.pb
I20260926 16:36:35.395622 28892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:36:35.400921 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 28825
I20260926 16:36:35.401113 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440233850748-6276-0/minicluster-data/master-2/wal/instance
I20260926 16:36:35.401571 6276 external_mini_cluster.cc:1428] Setting key f40ea702763ecd64919f9577a379f5a1
I20260926 16:36:35.402580 28892 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.33.62" port: 34579 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 16:36:35 dist-test-slave-khd3 krb5kdc[28664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440595, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.60@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
Sep 26 16:36:35 dist-test-slave-khd3 krb5kdc[28664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.60@KRBTEST.COM
I20260926 16:36:35.424109 28896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:35.405807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33668 (local address 127.6.33.62:34579)
0926 16:36:35.406440 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:35.406447 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:35.407124 (+ 677us) server_negotiation.cc:407] Connection header received
0926 16:36:35.407526 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:35.407532 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:35.407623 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:35.407738 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:35.409109 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:35.409905 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:35.411965 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:35.412579 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:35.412675 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:35.416692 (+ 4017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:35.416697 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:35.416700 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:36:35.416734 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:35.420292 (+ 3558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:35.421191 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:35.421197 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:35.421202 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:35.421364 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:35.421906 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:35.421909 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:35.421911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:35.422291 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:35.422453 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:35.423373 (+ 920us) server_negotiation.cc:299] Negotiation successful
0926 16:36:35.423652 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":338,"threads_started":1}
I20260926 16:36:35.425280 28895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:35.403584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45630 (local address 127.6.33.60:40133)
0926 16:36:35.404344 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:35.404357 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:35.404398 (+ 41us) server_negotiation.cc:407] Connection header received
0926 16:36:35.407178 (+ 2780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:35.407206 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:35.407491 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:35.407712 (+ 221us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:35.409257 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:35.410429 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:35.412364 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:35.413146 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:35.413262 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:35.417645 (+ 4383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:35.417661 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:35.417680 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:36:35.417723 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:35.421426 (+ 3703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:35.422224 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:35.422235 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:35.422239 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:35.422384 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:35.422859 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:35.422863 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:35.422865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:35.423418 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:35.423720 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:35.424064 (+ 344us) server_negotiation.cc:299] Negotiation successful
0926 16:36:35.424350 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":550,"spinlock_wait_cycles":32384,"thread_start_us":322,"threads_started":1}
I20260926 16:36:35.425320 28897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:35.405803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:34579 (local address 127.0.0.1:33668)
0926 16:36:35.406872 (+ 1069us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:35.406912 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 16:36:35.407287 (+ 375us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:36:35.407804 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:36:35.407811 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:36:35.408487 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:36:35.408936 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:36:35.408955 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:36:35.410084 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:36:35.410088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:36:35.411791 (+ 1703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:36:35.411811 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:36:35.412845 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:36:35.412852 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:36:35.413195 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:35.414057 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:36:35.414097 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:36:35.416439 (+ 2342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:36:35.420441 (+ 4002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:36:35.420451 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:36:35.420461 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:36:35.421070 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:36:35.421500 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:36:35.421507 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:36:35.421511 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:36:35.421769 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:36:35.422421 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:36:35.422449 (+ 28us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:36:35.422917 (+ 468us) client_negotiation.cc:769] Sending connection context
0926 16:36:35.423173 (+ 256us) client_negotiation.cc:240] Negotiation successful
0926 16:36:35.423674 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":730,"mutex_wait_us":1,"thread_start_us":569,"threads_started":1}
I20260926 16:36:35.427024 28892 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.33.61" port: 33249 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 16:36:35 dist-test-slave-khd3 krb5kdc[28664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440595, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.60@KRBTEST.COM for kudu/127.6.33.61@KRBTEST.COM
Sep 26 16:36:35 dist-test-slave-khd3 krb5kdc[28664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:36:35.443158 28822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:35.427844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48150 (local address 127.6.33.61:33249)
0926 16:36:35.428023 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:35.428028 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:35.428245 (+ 217us) server_negotiation.cc:407] Connection header received
0926 16:36:35.428401 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:35.428407 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:35.428521 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:35.428618 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:35.429785 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:35.430573 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:35.432031 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:35.432469 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:35.432569 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:35.436149 (+ 3580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:35.436154 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:35.436158 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:36:35.436194 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:35.439699 (+ 3505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:35.440429 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:35.440437 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:35.440442 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:35.440607 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:35.441278 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:35.441285 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:35.441290 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:35.441721 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:35.441893 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:35.442434 (+ 541us) server_negotiation.cc:299] Negotiation successful
0926 16:36:35.442697 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 16:36:35.443174 28897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:35.427862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.61:33249 (local address 127.0.0.1:48150)
0926 16:36:35.428141 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:35.428160 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 16:36:35.428263 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:36:35.428847 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:36:35.428853 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:36:35.429363 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:36:35.429599 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:36:35.429614 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:36:35.430747 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:36:35.430751 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:36:35.431881 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:36:35.431896 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:36:35.432625 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:36:35.432633 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:36:35.432865 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:35.433539 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:36:35.433565 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:36:35.435894 (+ 2329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:36:35.439878 (+ 3984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:36:35.439883 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:36:35.439886 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:36:35.440300 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:36:35.440824 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:36:35.440830 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:36:35.440833 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:36:35.441042 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:36:35.441874 (+ 832us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:36:35.441878 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:36:35.442086 (+ 208us) client_negotiation.cc:769] Sending connection context
0926 16:36:35.442249 (+ 163us) client_negotiation.cc:240] Negotiation successful
0926 16:36:35.442583 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172}
I20260926 16:36:35.444408 28892 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.33.60" port: 40133 } has no permanent_uuid. Determining permanent_uuid...
I20260926 16:36:35.445062 28896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:35.428494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33678 (local address 127.6.33.62:34579)
0926 16:36:35.428666 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:35.428670 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:35.428703 (+ 33us) server_negotiation.cc:407] Connection header received
0926 16:36:35.428953 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:35.428958 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:35.429033 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:35.429116 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:35.430496 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:35.431332 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:35.432892 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:35.433292 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:35.433387 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:35.437785 (+ 4398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:35.437795 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:35.437802 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:36:35.437854 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:35.441055 (+ 3201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:35.441956 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:35.441962 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:35.441965 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:35.442178 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:35.442652 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:35.442657 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:35.442661 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:35.443043 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:35.443356 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:35.444366 (+ 1010us) server_negotiation.cc:299] Negotiation successful
0926 16:36:35.444699 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Sep 26 16:36:35 dist-test-slave-khd3 krb5kdc[28664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440595, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.60@KRBTEST.COM for kudu/127.6.33.60@KRBTEST.COM
Sep 26 16:36:35 dist-test-slave-khd3 krb5kdc[28664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.61@KRBTEST.COM
I20260926 16:36:35.458894 28895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:35.445467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45642 (local address 127.6.33.60:40133)
0926 16:36:35.445622 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:35.445628 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:35.445647 (+ 19us) server_negotiation.cc:407] Connection header received
0926 16:36:35.445701 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:35.445708 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:35.445787 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:35.445891 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:35.446785 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:35.447603 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:35.448776 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:35.449175 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:35.449307 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:35.452580 (+ 3273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:35.452585 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:35.452588 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:36:35.452616 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:35.455690 (+ 3074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:35.456522 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:35.456527 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:35.456530 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:35.456730 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:35.457173 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:35.457178 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:35.457182 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:35.457587 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:35.457748 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:35.458157 (+ 409us) server_negotiation.cc:299] Negotiation successful
0926 16:36:35.458432 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 16:36:35.458895 28897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:35.445263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:40133 (local address 127.0.0.1:45642)
0926 16:36:35.445431 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:35.445450 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 16:36:35.445557 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:36:35.445926 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:36:35.445929 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:36:35.446470 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:36:35.446613 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:36:35.446622 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:36:35.447727 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:36:35.447734 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:36:35.448617 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:36:35.448628 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:36:35.449312 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:36:35.449317 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:36:35.449514 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:35.450150 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:36:35.450172 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:36:35.452374 (+ 2202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:36:35.455850 (+ 3476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:36:35.455857 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:36:35.455861 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:36:35.456404 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:36:35.456892 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:36:35.456896 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:36:35.456899 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:36:35.457042 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:36:35.457756 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:36:35.457759 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:36:35.457882 (+ 123us) client_negotiation.cc:769] Sending connection context
0926 16:36:35.458029 (+ 147us) client_negotiation.cc:240] Negotiation successful
0926 16:36:35.458574 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":211}
I20260926 16:36:35.463133 28822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:35.447070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48156 (local address 127.6.33.61:33249)
0926 16:36:35.447255 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:35.447259 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:35.447272 (+ 13us) server_negotiation.cc:407] Connection header received
0926 16:36:35.447315 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:35.447319 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:35.447370 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:35.447474 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:35.448683 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:35.449504 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:35.450874 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:35.451290 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:35.451418 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:35.456433 (+ 5015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:35.456438 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:35.456441 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:36:35.456504 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:35.460187 (+ 3683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:35.461088 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:35.461093 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:35.461095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:35.461235 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:35.461701 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:35.461705 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:35.461707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:35.462039 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:35.462252 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:35.462617 (+ 365us) server_negotiation.cc:299] Negotiation successful
0926 16:36:35.462783 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 16:36:35.464896 28892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c81bd5efa06746d382bb22825ef84fa4: Bootstrap starting.
I20260926 16:36:35.468077 6276 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:36:35.468343 28892 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c81bd5efa06746d382bb22825ef84fa4: Neither blocks nor log segments found. Creating new log.
I20260926 16:36:35.469470 28892 log.cc:824] T 00000000000000000000000000000000 P c81bd5efa06746d382bb22825ef84fa4: Log is configured to *not* fsync() on all Append() calls
I20260926 16:36:35.472332 28892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c81bd5efa06746d382bb22825ef84fa4: No bootstrap required, opened a new log
I20260926 16:36:35.473134 28744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.33.60" port: 40133 } attempt: 4
I20260926 16:36:35.470927 28719 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "592bb7ca183c44dab01abef89341ddf2"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33678
I20260926 16:36:35.481855 28892 raft_consensus.cc:348] T 00000000000000000000000000000000 P c81bd5efa06746d382bb22825ef84fa4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "592bb7ca183c44dab01abef89341ddf2" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 34579 } } peers { permanent_uuid: "d5e35546b8d6472cb69b6465c4e09e63" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 33249 } } peers { permanent_uuid: "c81bd5efa06746d382bb22825ef84fa4" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 40133 } }
I20260926 16:36:35.482293 28892 raft_consensus.cc:729] T 00000000000000000000000000000000 P c81bd5efa06746d382bb22825ef84fa4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c81bd5efa06746d382bb22825ef84fa4, State: Initialized, Role: FOLLOWER
Sep 26 16:36:35 dist-test-slave-khd3 krb5kdc[28664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440594, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.62@KRBTEST.COM for kudu/127.6.33.60@KRBTEST.COM
I20260926 16:36:35.482919 28892 consensus_queue.cc:260] T 00000000000000000000000000000000 P c81bd5efa06746d382bb22825ef84fa4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "592bb7ca183c44dab01abef89341ddf2" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 34579 } } peers { permanent_uuid: "d5e35546b8d6472cb69b6465c4e09e63" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 33249 } } peers { permanent_uuid: "c81bd5efa06746d382bb22825ef84fa4" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 40133 } }
I20260926 16:36:35.484063 28899 sys_catalog.cc:455] T 00000000000000000000000000000000 P c81bd5efa06746d382bb22825ef84fa4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "592bb7ca183c44dab01abef89341ddf2" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 34579 } } peers { permanent_uuid: "d5e35546b8d6472cb69b6465c4e09e63" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 33249 } } peers { permanent_uuid: "c81bd5efa06746d382bb22825ef84fa4" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 40133 } } }
I20260926 16:36:35.484519 28899 sys_catalog.cc:458] T 00000000000000000000000000000000 P c81bd5efa06746d382bb22825ef84fa4 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:36:35.485610 28892 sys_catalog.cc:564] T 00000000000000000000000000000000 P c81bd5efa06746d382bb22825ef84fa4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:36:35.491932 28748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:35.473802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:40133 (local address 127.0.0.1:45652)
0926 16:36:35.474053 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:35.474073 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 16:36:35.474225 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:36:35.474785 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:36:35.474789 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:36:35.475297 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:36:35.475515 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:36:35.475529 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:36:35.477075 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:36:35.477091 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:36:35.478594 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:36:35.478612 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:36:35.479672 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:36:35.479680 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:36:35.479891 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:35.480611 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:36:35.480633 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:36:35.483678 (+ 3045us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:36:35.488453 (+ 4775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:36:35.488459 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:36:35.488463 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:36:35.489098 (+ 635us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:36:35.489694 (+ 596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:36:35.489701 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:36:35.489707 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:36:35.489929 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:36:35.490835 (+ 906us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:36:35.490840 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:36:35.491033 (+ 193us) client_negotiation.cc:769] Sending connection context
0926 16:36:35.491190 (+ 157us) client_negotiation.cc:240] Negotiation successful
0926 16:36:35.491514 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260926 16:36:35.493045 28895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:35.474222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45652 (local address 127.6.33.60:40133)
0926 16:36:35.474426 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:35.474432 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:35.474452 (+ 20us) server_negotiation.cc:407] Connection header received
0926 16:36:35.474517 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:35.474533 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:35.474615 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:35.474722 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:35.475677 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:35.476893 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:35.478753 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:35.479513 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:35.479654 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:35.483922 (+ 4268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:35.483928 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:35.483936 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:36:35.483978 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:35.488217 (+ 4239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:35.489278 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:35.489285 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:35.489292 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:35.489499 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:35.490073 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:35.490081 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:35.490087 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:35.490680 (+ 593us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:35.490907 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:35.492325 (+ 1418us) server_negotiation.cc:299] Negotiation successful
0926 16:36:35.492574 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 16:36:35.499629 28744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 592bb7ca183c44dab01abef89341ddf2: Bootstrap starting.
W20260926 16:36:35.499799 28910 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c81bd5efa06746d382bb22825ef84fa4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 16:36:35.499909 28910 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 16:36:35.503659 28744 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 592bb7ca183c44dab01abef89341ddf2: Neither blocks nor log segments found. Creating new log.
I20260926 16:36:35.505017 28744 log.cc:824] T 00000000000000000000000000000000 P 592bb7ca183c44dab01abef89341ddf2: Log is configured to *not* fsync() on all Append() calls
I20260926 16:36:35.495111 28825 master_runner.cc:186] Error getting master registration for 127.6.33.62:34579: OK, instance_id { permanent_uuid: "592bb7ca183c44dab01abef89341ddf2" instance_seqno: 1790440594327289 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 16:36:35.506819 28825 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 16:36:35.508283 28744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 592bb7ca183c44dab01abef89341ddf2: No bootstrap required, opened a new log
I20260926 16:36:35.515161 28744 raft_consensus.cc:348] T 00000000000000000000000000000000 P 592bb7ca183c44dab01abef89341ddf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "592bb7ca183c44dab01abef89341ddf2" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 34579 } } peers { permanent_uuid: "d5e35546b8d6472cb69b6465c4e09e63" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 33249 } } peers { permanent_uuid: "c81bd5efa06746d382bb22825ef84fa4" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 40133 } }
I20260926 16:36:35.515609 28744 raft_consensus.cc:729] T 00000000000000000000000000000000 P 592bb7ca183c44dab01abef89341ddf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 592bb7ca183c44dab01abef89341ddf2, State: Initialized, Role: FOLLOWER
I20260926 16:36:35.516122 28744 consensus_queue.cc:260] T 00000000000000000000000000000000 P 592bb7ca183c44dab01abef89341ddf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "592bb7ca183c44dab01abef89341ddf2" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 34579 } } peers { permanent_uuid: "d5e35546b8d6472cb69b6465c4e09e63" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 33249 } } peers { permanent_uuid: "c81bd5efa06746d382bb22825ef84fa4" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 40133 } }
I20260926 16:36:35.517243 28913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 592bb7ca183c44dab01abef89341ddf2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "592bb7ca183c44dab01abef89341ddf2" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 34579 } } peers { permanent_uuid: "d5e35546b8d6472cb69b6465c4e09e63" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 33249 } } peers { permanent_uuid: "c81bd5efa06746d382bb22825ef84fa4" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 40133 } } }
I20260926 16:36:35.517618 28913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 592bb7ca183c44dab01abef89341ddf2 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:36:35.518515 28744 sys_catalog.cc:564] T 00000000000000000000000000000000 P 592bb7ca183c44dab01abef89341ddf2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:36:35.518620 28719 raft_consensus.cc:482] T 00000000000000000000000000000000 P 592bb7ca183c44dab01abef89341ddf2 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 16:36:35.518821 28719 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 592bb7ca183c44dab01abef89341ddf2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:36:35.523219 28719 raft_consensus.cc:504] T 00000000000000000000000000000000 P 592bb7ca183c44dab01abef89341ddf2 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "592bb7ca183c44dab01abef89341ddf2" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 34579 } } peers { permanent_uuid: "d5e35546b8d6472cb69b6465c4e09e63" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 33249 } } peers { permanent_uuid: "c81bd5efa06746d382bb22825ef84fa4" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 40133 } }
I20260926 16:36:35.526271 28719 leader_election.cc:288] T 00000000000000000000000000000000 P 592bb7ca183c44dab01abef89341ddf2 [CANDIDATE]: Term 1 election: Requested vote from peers d5e35546b8d6472cb69b6465c4e09e63 (127.6.33.61:33249), c81bd5efa06746d382bb22825ef84fa4 (127.6.33.60:40133)
I20260926 16:36:35.527127 28867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "592bb7ca183c44dab01abef89341ddf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c81bd5efa06746d382bb22825ef84fa4"
I20260926 16:36:35.527691 28867 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c81bd5efa06746d382bb22825ef84fa4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:36:35.532284 28867 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c81bd5efa06746d382bb22825ef84fa4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 592bb7ca183c44dab01abef89341ddf2 in term 1.
I20260926 16:36:35.533150 28694 leader_election.cc:302] T 00000000000000000000000000000000 P 592bb7ca183c44dab01abef89341ddf2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 592bb7ca183c44dab01abef89341ddf2, c81bd5efa06746d382bb22825ef84fa4; no voters:
I20260926 16:36:35.533797 28913 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 592bb7ca183c44dab01abef89341ddf2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:36:35.547529 28913 raft_consensus.cc:686] T 00000000000000000000000000000000 P 592bb7ca183c44dab01abef89341ddf2 [term 1 LEADER]: Becoming Leader. State: Replica: 592bb7ca183c44dab01abef89341ddf2, State: Running, Role: LEADER
I20260926 16:36:35.549059 28913 consensus_queue.cc:237] T 00000000000000000000000000000000 P 592bb7ca183c44dab01abef89341ddf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "592bb7ca183c44dab01abef89341ddf2" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 34579 } } peers { permanent_uuid: "d5e35546b8d6472cb69b6465c4e09e63" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 33249 } } peers { permanent_uuid: "c81bd5efa06746d382bb22825ef84fa4" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 40133 } }
I20260926 16:36:35.563934 28919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 592bb7ca183c44dab01abef89341ddf2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 592bb7ca183c44dab01abef89341ddf2. Latest consensus state: current_term: 1 leader_uuid: "592bb7ca183c44dab01abef89341ddf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "592bb7ca183c44dab01abef89341ddf2" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 34579 } } peers { permanent_uuid: "d5e35546b8d6472cb69b6465c4e09e63" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 33249 } } peers { permanent_uuid: "c81bd5efa06746d382bb22825ef84fa4" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 40133 } } }
I20260926 16:36:35.564949 28919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 592bb7ca183c44dab01abef89341ddf2 [sys.catalog]: This master's current role is: LEADER
W20260926 16:36:35.566241 28698 tablet.cc:2369] T 00000000000000000000000000000000 P 592bb7ca183c44dab01abef89341ddf2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:36:35.573292 28928 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:36:35.575682 28818 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.33.60" port: 40133 } attempt: 5
Sep 26 16:36:35 dist-test-slave-khd3 krb5kdc[28664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440594, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.61@KRBTEST.COM for kudu/127.6.33.60@KRBTEST.COM
I20260926 16:36:35.611545 28928 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:36:35.616132 28820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:35.576326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:40133 (local address 127.0.0.1:45664)
0926 16:36:35.576740 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:35.576781 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 16:36:35.576895 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:36:35.591604 (+ 14709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:36:35.591612 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:36:35.592136 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:36:35.592359 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:36:35.592370 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:36:35.595582 (+ 3212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:36:35.595590 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:36:35.597045 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:36:35.597062 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:36:35.598857 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:36:35.598864 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:36:35.599127 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:35.599908 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:36:35.599931 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:36:35.603014 (+ 3083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:36:35.612256 (+ 9242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:36:35.612274 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:36:35.612278 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:36:35.612899 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:36:35.613702 (+ 803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:36:35.613708 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:36:35.613712 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:36:35.613884 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:36:35.615070 (+ 1186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:36:35.615076 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:36:35.615250 (+ 174us) client_negotiation.cc:769] Sending connection context
0926 16:36:35.615389 (+ 139us) client_negotiation.cc:240] Negotiation successful
0926 16:36:35.615728 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":1}
I20260926 16:36:35.618005 28932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:35.577300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45664 (local address 127.6.33.60:40133)
0926 16:36:35.590998 (+ 13698us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:35.591006 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:35.591058 (+ 52us) server_negotiation.cc:407] Connection header received
0926 16:36:35.591148 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:35.591154 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:35.591322 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:35.592583 (+ 1261us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:35.593042 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:35.595345 (+ 2303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:35.597316 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:35.598603 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:35.603272 (+ 4669us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:35.603680 (+ 408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:35.603690 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:35.603695 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:36:35.603759 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:35.612003 (+ 8244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:35.613169 (+ 1166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:35.613176 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:35.613180 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:35.613491 (+ 311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:35.614167 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:35.614197 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:35.614200 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:35.614873 (+ 673us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:35.616644 (+ 1771us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:35.616868 (+ 224us) server_negotiation.cc:299] Negotiation successful
0926 16:36:35.617305 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13426,"thread_start_us":13026,"threads_started":1}
I20260926 16:36:35.618360 28822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:35.558200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48160 (local address 127.6.33.61:33249)
0926 16:36:35.558449 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:35.558456 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:35.568194 (+ 9738us) server_negotiation.cc:407] Connection header received
0926 16:36:35.568637 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:35.568646 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:35.568738 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:35.568843 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:35.582194 (+ 13351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:35.583547 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:35.591901 (+ 8354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:35.592659 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:35.592785 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:35.603900 (+ 11115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:35.603909 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:35.603916 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:36:35.603956 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:35.607492 (+ 3536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:35.609557 (+ 2065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:35.609567 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:35.609573 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:35.609750 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:35.616376 (+ 6626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:35.616382 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:35.616385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:35.616750 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:35.617033 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:35.617707 (+ 674us) server_negotiation.cc:299] Negotiation successful
0926 16:36:35.617960 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":78}
I20260926 16:36:35.622737 28896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:35.553619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33684 (local address 127.6.33.62:34579)
0926 16:36:35.553824 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:35.553830 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:35.567824 (+ 13994us) server_negotiation.cc:407] Connection header received
0926 16:36:35.570840 (+ 3016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:35.570848 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:35.570946 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:35.571023 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:35.577759 (+ 6736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:35.579193 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:35.589698 (+ 10505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:35.601201 (+ 11503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:35.601412 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:35.613004 (+ 11592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:35.613014 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:35.613023 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:36:35.613087 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:35.618060 (+ 4973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:35.619486 (+ 1426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:35.619496 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:35.619502 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:35.619711 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:35.620285 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:35.620293 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:35.620299 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:35.620790 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:35.621142 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:35.621940 (+ 798us) server_negotiation.cc:299] Negotiation successful
0926 16:36:35.622243 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260926 16:36:35.623091 28846 tablet.cc:2369] T 00000000000000000000000000000000 P c81bd5efa06746d382bb22825ef84fa4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:36:35.624403 28818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5e35546b8d6472cb69b6465c4e09e63: Bootstrap starting.
I20260926 16:36:35.626499 28895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:35.565698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45654 (local address 127.6.33.60:40133)
0926 16:36:35.565906 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:35.565913 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:35.582045 (+ 16132us) server_negotiation.cc:407] Connection header received
0926 16:36:35.588004 (+ 5959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:35.588013 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:35.588169 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:35.588265 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:35.593984 (+ 5719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:35.595121 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:35.599689 (+ 4568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:35.600445 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:35.600549 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:35.619548 (+ 18999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:35.619575 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:35.619580 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:36:35.619620 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:35.623171 (+ 3551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:35.624089 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:35.624094 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:35.624097 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:35.624246 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:35.624731 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:35.624739 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:35.624743 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:35.625139 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:35.625378 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:35.625906 (+ 528us) server_negotiation.cc:299] Negotiation successful
0926 16:36:35.626154 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 16:36:35.632934 28867 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c81bd5efa06746d382bb22825ef84fa4 [term 1 FOLLOWER]: Refusing update from remote peer 592bb7ca183c44dab01abef89341ddf2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:36:35.633944 28919 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 592bb7ca183c44dab01abef89341ddf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c81bd5efa06746d382bb22825ef84fa4" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 40133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:36:35.638607 28818 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5e35546b8d6472cb69b6465c4e09e63: Neither blocks nor log segments found. Creating new log.
I20260926 16:36:35.645504 28818 log.cc:824] T 00000000000000000000000000000000 P d5e35546b8d6472cb69b6465c4e09e63: Log is configured to *not* fsync() on all Append() calls
I20260926 16:36:35.649775 28939 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 16:36:35.658172 28818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5e35546b8d6472cb69b6465c4e09e63: No bootstrap required, opened a new log
I20260926 16:36:35.656204 28677 master_runner.cc:186] Error getting master registration for 127.6.33.61:33249: OK, instance_id { permanent_uuid: "d5e35546b8d6472cb69b6465c4e09e63" instance_seqno: 1790440594841830 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 16:36:35.668262 28677 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 16:36:35.677322 28899 sys_catalog.cc:455] T 00000000000000000000000000000000 P c81bd5efa06746d382bb22825ef84fa4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 592bb7ca183c44dab01abef89341ddf2. Latest consensus state: current_term: 1 leader_uuid: "592bb7ca183c44dab01abef89341ddf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "592bb7ca183c44dab01abef89341ddf2" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 34579 } } peers { permanent_uuid: "d5e35546b8d6472cb69b6465c4e09e63" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 33249 } } peers { permanent_uuid: "c81bd5efa06746d382bb22825ef84fa4" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 40133 } } }
I20260926 16:36:35.677644 28899 sys_catalog.cc:458] T 00000000000000000000000000000000 P c81bd5efa06746d382bb22825ef84fa4 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:36:35.680157 28934 mvcc.cc:205] Tried to move back new op lower bound from 7333644679699673088 to 7333644679384760320. Current Snapshot: MvccSnapshot[applied={T|T < 7333644679699673088}]
I20260926 16:36:35.682173 28936 mvcc.cc:205] Tried to move back new op lower bound from 7333644679699673088 to 7333644679384760320. Current Snapshot: MvccSnapshot[applied={T|T < 7333644679699673088}]
I20260926 16:36:35.689316 28899 sys_catalog.cc:455] T 00000000000000000000000000000000 P c81bd5efa06746d382bb22825ef84fa4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "592bb7ca183c44dab01abef89341ddf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "592bb7ca183c44dab01abef89341ddf2" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 34579 } } peers { permanent_uuid: "d5e35546b8d6472cb69b6465c4e09e63" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 33249 } } peers { permanent_uuid: "c81bd5efa06746d382bb22825ef84fa4" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 40133 } } }
I20260926 16:36:35.689594 28899 sys_catalog.cc:458] T 00000000000000000000000000000000 P c81bd5efa06746d382bb22825ef84fa4 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:36:35.693499 28818 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5e35546b8d6472cb69b6465c4e09e63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "592bb7ca183c44dab01abef89341ddf2" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 34579 } } peers { permanent_uuid: "d5e35546b8d6472cb69b6465c4e09e63" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 33249 } } peers { permanent_uuid: "c81bd5efa06746d382bb22825ef84fa4" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 40133 } }
I20260926 16:36:35.694661 28818 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5e35546b8d6472cb69b6465c4e09e63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5e35546b8d6472cb69b6465c4e09e63, State: Initialized, Role: FOLLOWER
I20260926 16:36:35.696169 28913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 592bb7ca183c44dab01abef89341ddf2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "592bb7ca183c44dab01abef89341ddf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "592bb7ca183c44dab01abef89341ddf2" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 34579 } } peers { permanent_uuid: "d5e35546b8d6472cb69b6465c4e09e63" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 33249 } } peers { permanent_uuid: "c81bd5efa06746d382bb22825ef84fa4" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 40133 } } }
I20260926 16:36:35.695727 28818 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5e35546b8d6472cb69b6465c4e09e63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "592bb7ca183c44dab01abef89341ddf2" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 34579 } } peers { permanent_uuid: "d5e35546b8d6472cb69b6465c4e09e63" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 33249 } } peers { permanent_uuid: "c81bd5efa06746d382bb22825ef84fa4" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 40133 } }
I20260926 16:36:35.696517 28913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 592bb7ca183c44dab01abef89341ddf2 [sys.catalog]: This master's current role is: LEADER
I20260926 16:36:35.697304 28928 catalog_manager.cc:1347] Generated new cluster ID: 43a55c6fdc3340769124e11f6a2ea967
I20260926 16:36:35.697417 28928 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:36:35.718827 28943 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5e35546b8d6472cb69b6465c4e09e63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "592bb7ca183c44dab01abef89341ddf2" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 34579 } } peers { permanent_uuid: "d5e35546b8d6472cb69b6465c4e09e63" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 33249 } } peers { permanent_uuid: "c81bd5efa06746d382bb22825ef84fa4" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 40133 } } }
I20260926 16:36:35.719503 28943 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5e35546b8d6472cb69b6465c4e09e63 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:36:35.721809 28792 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5e35546b8d6472cb69b6465c4e09e63 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:36:35.725028 28793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "592bb7ca183c44dab01abef89341ddf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d5e35546b8d6472cb69b6465c4e09e63"
I20260926 16:36:35.729358 28792 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d5e35546b8d6472cb69b6465c4e09e63 [term 1 FOLLOWER]: Refusing update from remote peer 592bb7ca183c44dab01abef89341ddf2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:36:35.732707 28793 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d5e35546b8d6472cb69b6465c4e09e63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 592bb7ca183c44dab01abef89341ddf2 in term 1.
I20260926 16:36:35.734242 28818 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5e35546b8d6472cb69b6465c4e09e63 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:36:35.734884 28919 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 592bb7ca183c44dab01abef89341ddf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5e35546b8d6472cb69b6465c4e09e63" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 33249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:36:35.742522 28928 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:36:35.746040 28928 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:36:35.772534 28928 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 592bb7ca183c44dab01abef89341ddf2: Generated new TSK 0
I20260926 16:36:35.773984 28928 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:36:35.776852 28943 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5e35546b8d6472cb69b6465c4e09e63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 592bb7ca183c44dab01abef89341ddf2. Latest consensus state: current_term: 1 leader_uuid: "592bb7ca183c44dab01abef89341ddf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "592bb7ca183c44dab01abef89341ddf2" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 34579 } } peers { permanent_uuid: "d5e35546b8d6472cb69b6465c4e09e63" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 33249 } } peers { permanent_uuid: "c81bd5efa06746d382bb22825ef84fa4" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 40133 } } }
I20260926 16:36:35.776777 28947 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5e35546b8d6472cb69b6465c4e09e63 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "592bb7ca183c44dab01abef89341ddf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "592bb7ca183c44dab01abef89341ddf2" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 34579 } } peers { permanent_uuid: "d5e35546b8d6472cb69b6465c4e09e63" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 33249 } } peers { permanent_uuid: "c81bd5efa06746d382bb22825ef84fa4" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 40133 } } }
I20260926 16:36:35.777163 28943 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5e35546b8d6472cb69b6465c4e09e63 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:36:35.777186 28947 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5e35546b8d6472cb69b6465c4e09e63 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:36:35.778355 28913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 592bb7ca183c44dab01abef89341ddf2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "592bb7ca183c44dab01abef89341ddf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "592bb7ca183c44dab01abef89341ddf2" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 34579 } } peers { permanent_uuid: "d5e35546b8d6472cb69b6465c4e09e63" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 33249 } } peers { permanent_uuid: "c81bd5efa06746d382bb22825ef84fa4" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 40133 } } }
I20260926 16:36:35.778671 28913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 592bb7ca183c44dab01abef89341ddf2 [sys.catalog]: This master's current role is: LEADER
I20260926 16:36:35.878311 28961 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260926 16:36:35.901512 6276 mini_kdc.cc:348] Time spent kdestroy: real 0.104s user 0.003s sys 0.025s
W20260926 16:36:35.914535 28896 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:36:35.905924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33690 (local address 127.6.33.62:34579)
0926 16:36:35.906214 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:35.906222 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:35.906246 (+ 24us) server_negotiation.cc:407] Connection header received
0926 16:36:35.906338 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:35.906342 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:35.906415 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:35.906528 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:35.914154 (+ 7626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33690: BlockingRecv error: recv got EOF from 127.0.0.1:33690 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
W20260926 16:36:35.918650 28895 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:36:35.907850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45678 (local address 127.6.33.60:40133)
0926 16:36:35.908022 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:35.908029 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:35.915861 (+ 7832us) server_negotiation.cc:407] Connection header received
0926 16:36:35.916884 (+ 1023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:35.916893 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:35.917025 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:35.917120 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:35.918313 (+ 1193us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45678: BlockingRecv error: recv got EOF from 127.0.0.1:45678 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260926 16:36:35.921016 28822 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:36:35.907716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48170 (local address 127.6.33.61:33249)
0926 16:36:35.907899 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:35.907905 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:35.918151 (+ 10246us) server_negotiation.cc:407] Connection header received
0926 16:36:35.918927 (+ 776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:35.918934 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:35.919024 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:35.919113 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:35.920667 (+ 1554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48170: BlockingRecv error: recv got EOF from 127.0.0.1:48170 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260926 16:36:35.929656 28822 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:36:35.927336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48182 (local address 127.6.33.61:33249)
0926 16:36:35.927524 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:35.927528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:35.928111 (+ 583us) server_negotiation.cc:407] Connection header received
0926 16:36:35.928383 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:35.928387 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:35.928495 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:35.928611 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:35.929368 (+ 757us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48182: BlockingRecv error: recv got EOF from 127.0.0.1:48182 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260926 16:36:35.938678 28822 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:36:35.934100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48184 (local address 127.6.33.61:33249)
0926 16:36:35.934274 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:35.934279 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:35.937324 (+ 3045us) server_negotiation.cc:407] Connection header received
0926 16:36:35.937555 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:35.937560 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:35.937632 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:35.937740 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:35.938501 (+ 761us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48184: BlockingRecv error: recv got EOF from 127.0.0.1:48184 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260926 16:36:35.946604 28822 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:36:35.944462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48194 (local address 127.6.33.61:33249)
0926 16:36:35.944691 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:35.944700 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:35.945220 (+ 520us) server_negotiation.cc:407] Connection header received
0926 16:36:35.945434 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:35.945439 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:35.945530 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:35.945640 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:35.946356 (+ 716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48194: BlockingRecv error: recv got EOF from 127.0.0.1:48194 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260926 16:36:35.955974 28822 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:36:35.953756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48196 (local address 127.6.33.61:33249)
0926 16:36:35.954081 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:35.954090 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:35.954628 (+ 538us) server_negotiation.cc:407] Connection header received
0926 16:36:35.954849 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:35.954855 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:35.954939 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:35.955046 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:35.955736 (+ 690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48196: BlockingRecv error: recv got EOF from 127.0.0.1:48196 (error 108)
Metrics: {"server-negotiator.queue_time_us":137}
W20260926 16:36:35.972097 28822 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:36:35.969756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48208 (local address 127.6.33.61:33249)
0926 16:36:35.970000 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:35.970010 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:35.970555 (+ 545us) server_negotiation.cc:407] Connection header received
0926 16:36:35.970786 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:35.970792 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:35.970891 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:35.970992 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:35.971809 (+ 817us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48208: BlockingRecv error: recv got EOF from 127.0.0.1:48208 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260926 16:36:36.005374 28822 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:36:36.002872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48214 (local address 127.6.33.61:33249)
0926 16:36:36.003067 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:36.003074 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:36.003621 (+ 547us) server_negotiation.cc:407] Connection header received
0926 16:36:36.003859 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:36.003866 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:36.003983 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:36.004119 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:36.005093 (+ 974us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48214: BlockingRecv error: recv got EOF from 127.0.0.1:48214 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260926 16:36:36.045540 28822 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:36:36.043001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48228 (local address 127.6.33.61:33249)
0926 16:36:36.043214 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:36.043222 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:36.043864 (+ 642us) server_negotiation.cc:407] Connection header received
0926 16:36:36.044090 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:36.044097 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:36.044200 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:36.044327 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:36.045271 (+ 944us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48228: BlockingRecv error: recv got EOF from 127.0.0.1:48228 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20260926 16:36:36.116904 28822 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:36:36.114851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48238 (local address 127.6.33.61:33249)
0926 16:36:36.115047 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:36.115053 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:36.115449 (+ 396us) server_negotiation.cc:407] Connection header received
0926 16:36:36.115637 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:36.115643 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:36.115740 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:36.115879 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:36.116655 (+ 776us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48238: BlockingRecv error: recv got EOF from 127.0.0.1:48238 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260926 16:36:36.252655 28822 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:36:36.250372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48240 (local address 127.6.33.61:33249)
0926 16:36:36.250626 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:36.250635 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:36.250941 (+ 306us) server_negotiation.cc:407] Connection header received
0926 16:36:36.251248 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:36.251257 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:36.251381 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:36.251508 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:36.252381 (+ 873us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48240: BlockingRecv error: recv got EOF from 127.0.0.1:48240 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 16:36:36.501977 28910 catalog_manager.cc:1259] Loaded cluster ID: 43a55c6fdc3340769124e11f6a2ea967
I20260926 16:36:36.502116 28910 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c81bd5efa06746d382bb22825ef84fa4: loading cluster ID for follower catalog manager: success
I20260926 16:36:36.504608 28910 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c81bd5efa06746d382bb22825ef84fa4: acquiring CA information for follower catalog manager: success
I20260926 16:36:36.506646 28910 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c81bd5efa06746d382bb22825ef84fa4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 16:36:36.514537 28822 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:36:36.512501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40620 (local address 127.6.33.61:33249)
0926 16:36:36.512773 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:36.512780 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:36.512992 (+ 212us) server_negotiation.cc:407] Connection header received
0926 16:36:36.513256 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:36.513260 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:36.513352 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:36.513463 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:36.514305 (+ 842us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40620: BlockingRecv error: recv got EOF from 127.0.0.1:40620 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 16:36:36.883083 28961 catalog_manager.cc:1259] Loaded cluster ID: 43a55c6fdc3340769124e11f6a2ea967
I20260926 16:36:36.883253 28961 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d5e35546b8d6472cb69b6465c4e09e63: loading cluster ID for follower catalog manager: success
I20260926 16:36:36.886271 28961 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d5e35546b8d6472cb69b6465c4e09e63: acquiring CA information for follower catalog manager: success
I20260926 16:36:36.888726 28961 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d5e35546b8d6472cb69b6465c4e09e63: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 16:36:37.041412 29025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:37.033322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40632 (local address 127.6.33.61:33249)
0926 16:36:37.034147 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:37.034153 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:37.034441 (+ 288us) server_negotiation.cc:407] Connection header received
0926 16:36:37.034673 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:37.034677 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:37.034860 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:37.035037 (+ 177us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:36:37.035643 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:37.036516 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:37.038653 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:37.039194 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:37.039303 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:37.039910 (+ 607us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:36:37.040588 (+ 678us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:36:37.040708 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:37.040861 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 16:36:37.040940 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":403,"threads_started":1}
I20260926 16:36:37.059302 29031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:37.052665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43008 (local address 127.6.33.60:40133)
0926 16:36:37.053514 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:37.053522 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:37.053550 (+ 28us) server_negotiation.cc:407] Connection header received
0926 16:36:37.053681 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:37.053685 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:37.053845 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:37.054038 (+ 193us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:36:37.054489 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:37.055238 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:37.056719 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:37.057290 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:37.057430 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:37.057972 (+ 542us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:36:37.058587 (+ 615us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:36:37.058775 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:37.058860 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 16:36:37.058946 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":726,"thread_start_us":600,"threads_started":1}
I20260926 16:36:37.082820 29025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:37.066599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40648 (local address 127.6.33.61:33249)
0926 16:36:37.066792 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:37.066798 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:37.074595 (+ 7797us) server_negotiation.cc:407] Connection header received
0926 16:36:37.074986 (+ 391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:37.074997 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:37.075124 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:37.075210 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:36:37.076911 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:37.078133 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:37.080512 (+ 2379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:37.081205 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:37.081327 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:37.081939 (+ 612us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:36:37.082092 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:36:37.082164 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:37.082402 (+ 238us) server_negotiation.cc:299] Negotiation successful
0926 16:36:37.082498 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 16:36:37.083667 29031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:37.067834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43012 (local address 127.6.33.60:40133)
0926 16:36:37.068040 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:37.068048 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:37.076999 (+ 8951us) server_negotiation.cc:407] Connection header received
0926 16:36:37.077293 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:37.077302 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:37.077394 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:37.077471 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:36:37.078210 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:37.079245 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:37.081543 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:37.082162 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:37.082267 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:37.082900 (+ 633us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:36:37.083035 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:36:37.083097 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:37.083303 (+ 206us) server_negotiation.cc:299] Negotiation successful
0926 16:36:37.083385 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 16:36:37.084481 29038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:37.066777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44162 (local address 127.6.33.62:34579)
0926 16:36:37.076534 (+ 9757us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:37.076543 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:37.076566 (+ 23us) server_negotiation.cc:407] Connection header received
0926 16:36:37.076689 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:37.076694 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:37.077000 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:37.077774 (+ 774us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:36:37.077965 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:37.079065 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:37.081575 (+ 2510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:37.082185 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:37.082859 (+ 674us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:37.083054 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:36:37.083533 (+ 479us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:36:37.084039 (+ 506us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:37.084092 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 16:36:37.084174 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9605,"thread_start_us":9504,"threads_started":1}
I20260926 16:36:37.090319 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 28677
I20260926 16:36:37.091468 28739 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:36:37.206303 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 28677
I20260926 16:36:37.229532 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 28751
I20260926 16:36:37.230830 28813 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:36:37.351241 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 28751
I20260926 16:36:37.371291 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 28825
I20260926 16:36:37.372493 28887 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:36:37.481441 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 28825
2026-09-26T16:36:37Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5011 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:36:37 dist-test-slave-khd3 krb5kdc[29045](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:36:37 dist-test-slave-khd3 krb5kdc[29045](info): set up 2 sockets
Sep 26 16:36:37 dist-test-slave-khd3 krb5kdc[29045](info): commencing operation
krb5kdc: starting...
W20260926 16:36:38.602416 6276 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/krb5kdc/test-user.keytab.
Sep 26 16:36:38 dist-test-slave-khd3 krb5kdc[29045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:36:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:36:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab.
Entry for principal kudu/127.6.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab.
I20260926 16:36:38.870311 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:45999
--webserver_interface=127.6.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:33859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.33.62:45999,127.6.33.61:35415,127.6.33.60:39025
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:36:39.212666 29058 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 16:36:39.213162 29058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:36:39.213284 29058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:36:39.213438 29058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:36:39.221283 29058 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:36:39.221428 29058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:36:39.221753 29058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:36:39.221885 29058 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:36:39.222007 29058 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:36:39.232684 29058 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:33859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.33.62:45999,127.6.33.61:35415,127.6.33.60:39025
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.62.keytab
--principal=kudu/127.6.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.62:45999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:36:39.234119 29058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:36:39.235613 29058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:36:39.248214 29063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:36:39.248304 29064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:36:39.248317 29066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:36:39.254778 29058 server_base.cc:1034] running on GCE node
I20260926 16:36:39.255716 29058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:36:39.257408 29058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:36:39.258674 29058 hybrid_clock.cc:648] HybridClock initialized: now 1790440599258612 us; error 44 us; skew 500 ppm
I20260926 16:36:39.259028 29058 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:36:39 dist-test-slave-khd3 krb5kdc[29045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440599, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:36:39.262480 29058 init.cc:377] Logged in from keytab as kudu/127.6.33.62@KRBTEST.COM (short username kudu)
I20260926 16:36:39.264209 29058 webserver.cc:460] Webserver started at http://127.6.33.62:39957/ using document root <none> and password file <none>
I20260926 16:36:39.265465 29058 fs_manager.cc:359] Metadata directory not provided
I20260926 16:36:39.265628 29058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:36:39.265985 29058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:36:39.269146 29058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/minicluster-data/master-0/data/instance:
uuid: "3b64decc87f94872a607612ef8fc2a33"
format_stamp: "Formatted at 2026-09-26 16:36:39 on dist-test-slave-khd3"
server_key: "dbc4be34a1642bc69e45cf213aa4c028"
server_key_iv: "d0c7dbcdeeda6076176675050e7d11ad"
server_key_version: "encryptionkey@0"
I20260926 16:36:39.270035 29058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/minicluster-data/master-0/wal/instance:
uuid: "3b64decc87f94872a607612ef8fc2a33"
format_stamp: "Formatted at 2026-09-26 16:36:39 on dist-test-slave-khd3"
server_key: "dbc4be34a1642bc69e45cf213aa4c028"
server_key_iv: "d0c7dbcdeeda6076176675050e7d11ad"
server_key_version: "encryptionkey@0"
I20260926 16:36:39.275688 29058 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 16:36:39.279340 29073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:36:39.280803 29058 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260926 16:36:39.280992 29058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/minicluster-data/master-0/wal
uuid: "3b64decc87f94872a607612ef8fc2a33"
format_stamp: "Formatted at 2026-09-26 16:36:39 on dist-test-slave-khd3"
server_key: "dbc4be34a1642bc69e45cf213aa4c028"
server_key_iv: "d0c7dbcdeeda6076176675050e7d11ad"
server_key_version: "encryptionkey@0"
I20260926 16:36:39.281236 29058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:36:39.303505 29058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:36:39.308228 29058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:36:39.308527 29058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:36:39.327879 29058 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.62:45999
I20260926 16:36:39.327898 29124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.62:45999 every 8 connection(s)
I20260926 16:36:39.329389 29058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/minicluster-data/master-0/data/info.pb
I20260926 16:36:39.333456 29125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:36:39.337563 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 29058
I20260926 16:36:39.337757 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/minicluster-data/master-0/wal/instance
I20260926 16:36:39.338299 6276 external_mini_cluster.cc:1428] Setting key f1ee941e8b4e01ecb46fe50b108eea02
I20260926 16:36:39.341949 29125 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.33.62" port: 45999 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 16:36:39 dist-test-slave-khd3 krb5kdc[29045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440599, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.62@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.33.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.61@KRBTEST.COM" created.
I20260926 16:36:39.369287 29128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:39.346022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:45999 (local address 127.0.0.1:55584)
0926 16:36:39.347250 (+ 1228us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:39.347297 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 16:36:39.349611 (+ 2314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:36:39.350661 (+ 1050us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:36:39.350666 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:36:39.351242 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:36:39.351591 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:36:39.351607 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:36:39.354014 (+ 2407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:36:39.354019 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:36:39.355128 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:36:39.355140 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:36:39.356519 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:36:39.356531 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:36:39.356689 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:39.357592 (+ 903us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:36:39.357618 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:36:39.359864 (+ 2246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:36:39.364672 (+ 4808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:36:39.364683 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:36:39.364698 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:36:39.365201 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:36:39.365719 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:36:39.365723 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:36:39.365726 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:36:39.365896 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:36:39.367008 (+ 1112us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:36:39.367015 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:36:39.367257 (+ 242us) client_negotiation.cc:769] Sending connection context
0926 16:36:39.367511 (+ 254us) client_negotiation.cc:240] Negotiation successful
0926 16:36:39.368127 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":945,"mutex_wait_us":110,"spinlock_wait_cycles":38144,"thread_start_us":704,"threads_started":1}
I20260926 16:36:39.369313 29127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:39.346037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55584 (local address 127.6.33.62:45999)
0926 16:36:39.346948 (+ 911us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:39.346966 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:39.347433 (+ 467us) server_negotiation.cc:407] Connection header received
0926 16:36:39.350011 (+ 2578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:39.350022 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:39.350411 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:39.350628 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:39.351800 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:39.353786 (+ 1986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:39.355286 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:39.356273 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:39.356469 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:39.360136 (+ 3667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:39.360150 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:39.360170 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:36:39.360216 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:39.364512 (+ 4296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:39.365344 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:39.365355 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:39.365361 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:39.365560 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:39.366040 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:39.366049 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:39.366063 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:39.366820 (+ 757us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:39.367027 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:39.367705 (+ 678us) server_negotiation.cc:299] Negotiation successful
0926 16:36:39.367958 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":648,"thread_start_us":363,"threads_started":1}
I20260926 16:36:39.372399 29125 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.33.61" port: 35415 } has no permanent_uuid. Determining permanent_uuid...
I20260926 16:36:39.373682 29128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:39.373155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.61:35415 (local address 127.0.0.1:59230)
0926 16:36:39.373366 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:39.373513 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.61:35415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260926 16:36:39.373944 29074 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.33.61:35415: connect: Connection refused (error 111)
I20260926 16:36:39.375587 29128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:39.375108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.61:35415 (local address 127.0.0.1:59246)
0926 16:36:39.375283 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:39.375393 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.61:35415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 16:36:39.375805 29125 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.33.61:35415: Network error: Client connection negotiation failed: client connection to 127.6.33.61:35415: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.61.keytab.
Entry for principal kudu/127.6.33.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.61.keytab.
I20260926 16:36:39.401746 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.61:35415
--webserver_interface=127.6.33.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.61.keytab
--principal=kudu/127.6.33.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:33859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.33.62:45999,127.6.33.61:35415,127.6.33.60:39025
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:36:39.425201 29125 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.33.61" port: 35415 } attempt: 1
I20260926 16:36:39.426525 29128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:39.426061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.61:35415 (local address 127.0.0.1:59256)
0926 16:36:39.426291 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:39.426396 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.61:35415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 16:36:39.427598 29128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:39.427235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.61:35415 (local address 127.0.0.1:59264)
0926 16:36:39.427382 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:39.427457 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.61:35415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 16:36:39.427773 29125 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.33.61:35415: Network error: Client connection negotiation failed: client connection to 127.6.33.61:35415: connect: Connection refused (error 111)
I20260926 16:36:39.496071 29125 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.33.61" port: 35415 } attempt: 2
I20260926 16:36:39.497371 29128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:39.496860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.61:35415 (local address 127.0.0.1:59272)
0926 16:36:39.497144 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:39.497239 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.61:35415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260926 16:36:39.498435 29128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:39.498081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.61:35415 (local address 127.0.0.1:59284)
0926 16:36:39.498249 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:39.498314 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.61:35415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 16:36:39.498621 29125 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.33.61:35415: Network error: Client connection negotiation failed: client connection to 127.6.33.61:35415: connect: Connection refused (error 111)
I20260926 16:36:39.589890 29125 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.33.61" port: 35415 } attempt: 3
I20260926 16:36:39.591078 29128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:39.590601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.61:35415 (local address 127.0.0.1:59286)
0926 16:36:39.590809 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:39.590926 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.61:35415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260926 16:36:39.592319 29128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:39.591818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.61:35415 (local address 127.0.0.1:59290)
0926 16:36:39.592027 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:39.592127 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.61:35415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260926 16:36:39.592588 29125 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.33.61:35415: Network error: Client connection negotiation failed: client connection to 127.6.33.61:35415: connect: Connection refused (error 111)
I20260926 16:36:39.735946 29125 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.33.61" port: 35415 } attempt: 4
I20260926 16:36:39.737529 29128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:39.736845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.61:35415 (local address 127.0.0.1:59306)
0926 16:36:39.737235 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:39.737343 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.61:35415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228}
I20260926 16:36:39.738665 29128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:39.738229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.61:35415 (local address 127.0.0.1:59322)
0926 16:36:39.738404 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:39.738512 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.61:35415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260926 16:36:39.738917 29125 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.33.61:35415: Network error: Client connection negotiation failed: client connection to 127.6.33.61:35415: connect: Connection refused (error 111)
W20260926 16:36:39.745637 29132 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 16:36:39.746181 29132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:36:39.746312 29132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:36:39.746443 29132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:36:39.756048 29132 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:36:39.756194 29132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:36:39.756297 29132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:36:39.756389 29132 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:36:39.756477 29132 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:36:39.766868 29132 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:33859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.33.62:45999,127.6.33.61:35415,127.6.33.60:39025
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.61.keytab
--principal=kudu/127.6.33.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.61:35415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.33.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:36:39.768282 29132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:36:39.769759 29132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:36:39.782214 29137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:36:39.782363 29138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:36:39.782385 29140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:36:39.783069 29132 server_base.cc:1034] running on GCE node
I20260926 16:36:39.783869 29132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:36:39.785537 29132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:36:39.786795 29132 hybrid_clock.cc:648] HybridClock initialized: now 1790440599786728 us; error 37 us; skew 500 ppm
I20260926 16:36:39.787232 29132 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:36:39 dist-test-slave-khd3 krb5kdc[29045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440599, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:36:39.790705 29132 init.cc:377] Logged in from keytab as kudu/127.6.33.61@KRBTEST.COM (short username kudu)
I20260926 16:36:39.792431 29132 webserver.cc:460] Webserver started at http://127.6.33.61:41747/ using document root <none> and password file <none>
I20260926 16:36:39.793277 29132 fs_manager.cc:359] Metadata directory not provided
I20260926 16:36:39.793393 29132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:36:39.793752 29132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:36:39.796954 29132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/minicluster-data/master-1/data/instance:
uuid: "4255e94899314359b96868d43dea1017"
format_stamp: "Formatted at 2026-09-26 16:36:39 on dist-test-slave-khd3"
server_key: "4d61d157cd433064042ad8c5d7883301"
server_key_iv: "d9d52785c60b135bf50954d136ae8e4a"
server_key_version: "encryptionkey@0"
I20260926 16:36:39.797811 29132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/minicluster-data/master-1/wal/instance:
uuid: "4255e94899314359b96868d43dea1017"
format_stamp: "Formatted at 2026-09-26 16:36:39 on dist-test-slave-khd3"
server_key: "4d61d157cd433064042ad8c5d7883301"
server_key_iv: "d9d52785c60b135bf50954d136ae8e4a"
server_key_version: "encryptionkey@0"
I20260926 16:36:39.803468 29132 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 16:36:39.807137 29147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:36:39.808723 29132 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 16:36:39.808995 29132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/minicluster-data/master-1/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/minicluster-data/master-1/wal
uuid: "4255e94899314359b96868d43dea1017"
format_stamp: "Formatted at 2026-09-26 16:36:39 on dist-test-slave-khd3"
server_key: "4d61d157cd433064042ad8c5d7883301"
server_key_iv: "d9d52785c60b135bf50954d136ae8e4a"
server_key_version: "encryptionkey@0"
I20260926 16:36:39.809247 29132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:36:39.839550 29132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:36:39.844317 29132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:36:39.844626 29132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:36:39.863938 29132 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.61:35415
I20260926 16:36:39.863994 29198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.61:35415 every 8 connection(s)
I20260926 16:36:39.865352 29132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/minicluster-data/master-1/data/info.pb
I20260926 16:36:39.868713 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 29132
I20260926 16:36:39.868928 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/minicluster-data/master-1/wal/instance
I20260926 16:36:39.869308 29199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:36:39.869426 6276 external_mini_cluster.cc:1428] Setting key 674bfb7de7691a4e2e00f2effda2192b
I20260926 16:36:39.875976 29199 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.33.62" port: 45999 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 16:36:39 dist-test-slave-khd3 krb5kdc[29045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440599, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.61@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.33.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.33.60@KRBTEST.COM" created.
I20260926 16:36:39.899833 29202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:39.878759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55586 (local address 127.6.33.62:45999)
0926 16:36:39.879328 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:39.879333 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:39.880439 (+ 1106us) server_negotiation.cc:407] Connection header received
0926 16:36:39.882505 (+ 2066us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:39.882511 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:39.882606 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:39.882731 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:39.884450 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:39.885293 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:39.887936 (+ 2643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:39.888373 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:39.888545 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:39.893107 (+ 4562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:39.893112 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:39.893115 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:36:39.893160 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:39.896279 (+ 3119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:39.897342 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:39.897346 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:39.897348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:39.897474 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:39.897981 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:39.897984 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:39.897986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:39.898283 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:39.898398 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:39.899389 (+ 991us) server_negotiation.cc:299] Negotiation successful
0926 16:36:39.899569 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":314,"threads_started":1}
I20260926 16:36:39.901405 29203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:39.879366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:45999 (local address 127.0.0.1:55586)
0926 16:36:39.880241 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:39.880287 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 16:36:39.882178 (+ 1891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:36:39.882998 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:36:39.883007 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:36:39.883575 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:36:39.884167 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:36:39.884184 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:36:39.885526 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:36:39.885534 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:36:39.887744 (+ 2210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:36:39.887763 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:36:39.888670 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:36:39.888679 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:36:39.889059 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:39.889909 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:36:39.889951 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:36:39.892850 (+ 2899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:36:39.896435 (+ 3585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:36:39.896446 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:36:39.896465 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:36:39.897202 (+ 737us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:36:39.897596 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:36:39.897604 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:36:39.897611 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:36:39.897864 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:36:39.898413 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:36:39.898425 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:36:39.898943 (+ 518us) client_negotiation.cc:769] Sending connection context
0926 16:36:39.899277 (+ 334us) client_negotiation.cc:240] Negotiation successful
0926 16:36:39.899856 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":389,"threads_started":1}
I20260926 16:36:39.903205 29199 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.33.61" port: 35415 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 16:36:39 dist-test-slave-khd3 krb5kdc[29045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440599, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.61@KRBTEST.COM for kudu/127.6.33.61@KRBTEST.COM
I20260926 16:36:39.918143 29203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:39.904010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.61:35415 (local address 127.0.0.1:59332)
0926 16:36:39.904235 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:39.904256 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 16:36:39.904361 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:36:39.905241 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:36:39.905252 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:36:39.905761 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:36:39.905987 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:36:39.906001 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:36:39.907400 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:36:39.907406 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:36:39.908252 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:36:39.908264 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:36:39.909165 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:36:39.909171 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:36:39.909389 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:39.909999 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:36:39.910016 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:36:39.911979 (+ 1963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:36:39.915434 (+ 3455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:36:39.915439 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:36:39.915443 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:36:39.915824 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:36:39.916214 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:36:39.916219 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:36:39.916223 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:36:39.916346 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:36:39.917085 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:36:39.917092 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:36:39.917235 (+ 143us) client_negotiation.cc:769] Sending connection context
0926 16:36:39.917393 (+ 158us) client_negotiation.cc:240] Negotiation successful
0926 16:36:39.917714 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"mutex_wait_us":5}
I20260926 16:36:39.918186 29204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:39.904148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59332 (local address 127.6.33.61:35415)
0926 16:36:39.904819 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:39.904826 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:39.904852 (+ 26us) server_negotiation.cc:407] Connection header received
0926 16:36:39.904896 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:39.904902 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:39.905084 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:39.905201 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:39.906176 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:39.907248 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:39.908414 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:39.909022 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:39.909165 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:39.912203 (+ 3038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:39.912215 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:39.912230 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:36:39.912279 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:39.915273 (+ 2994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:39.915959 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:39.915975 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:39.915977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:39.916097 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:39.916451 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:39.916458 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:39.916461 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:39.916905 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:39.917168 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:39.917540 (+ 372us) server_negotiation.cc:299] Negotiation successful
0926 16:36:39.917825 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":98,"server-negotiator.queue_time_us":466,"thread_start_us":316,"threads_started":1}
I20260926 16:36:39.920666 29199 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.33.60" port: 39025 } has no permanent_uuid. Determining permanent_uuid...
I20260926 16:36:39.922125 29203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:39.921631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:39025 (local address 127.0.0.1:56044)
0926 16:36:39.921811 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:39.921953 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.60:39025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260926 16:36:39.922508 29150 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.33.60:39025: connect: Connection refused (error 111)
I20260926 16:36:39.924155 29203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:39.923774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:39025 (local address 127.0.0.1:56056)
0926 16:36:39.923958 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:39.924035 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.60:39025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 16:36:39.924381 29199 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.33.60:39025: Network error: Client connection negotiation failed: client connection to 127.6.33.60:39025: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.33.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.60.keytab.
Entry for principal kudu/127.6.33.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.60.keytab.
I20260926 16:36:39.932713 6276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxavWi/build/asan/bin/kudu
/tmp/dist-test-taskVxavWi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.60:39025
--webserver_interface=127.6.33.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.60.keytab
--principal=kudu/127.6.33.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.33.20:33859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.33.62:45999,127.6.33.61:35415,127.6.33.60:39025
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:36:39.973707 29199 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.33.60" port: 39025 } attempt: 1
I20260926 16:36:39.974936 29203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:39.974455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:39025 (local address 127.0.0.1:56072)
0926 16:36:39.974675 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:39.974777 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.60:39025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260926 16:36:39.976362 29203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:39.975820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:39025 (local address 127.0.0.1:56080)
0926 16:36:39.976062 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:39.976176 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.60:39025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260926 16:36:39.976600 29199 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.33.60:39025: Network error: Client connection negotiation failed: client connection to 127.6.33.60:39025: connect: Connection refused (error 111)
I20260926 16:36:40.038276 29125 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.33.61" port: 35415 } attempt: 5
I20260926 16:36:40.044829 29199 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.33.60" port: 39025 } attempt: 2
I20260926 16:36:40.045900 29203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:40.045410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:39025 (local address 127.0.0.1:56086)
0926 16:36:40.045624 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:40.045722 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.60:39025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260926 16:36:40.046977 29203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:40.046645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:39025 (local address 127.0.0.1:56092)
0926 16:36:40.046797 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:40.046866 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.60:39025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
Sep 26 16:36:40 dist-test-slave-khd3 krb5kdc[29045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440599, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.62@KRBTEST.COM for kudu/127.6.33.61@KRBTEST.COM
W20260926 16:36:40.047222 29199 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.33.60:39025: Network error: Client connection negotiation failed: client connection to 127.6.33.60:39025: connect: Connection refused (error 111)
I20260926 16:36:40.054731 29128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:40.039097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.61:35415 (local address 127.0.0.1:59344)
0926 16:36:40.039323 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:40.039346 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 16:36:40.039493 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:36:40.039902 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:36:40.039906 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:36:40.040329 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:36:40.040542 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:36:40.040551 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:36:40.042058 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:36:40.042064 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:36:40.043222 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:36:40.043235 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:36:40.044243 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:36:40.044248 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:36:40.044390 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:40.045251 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:36:40.045278 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:36:40.047867 (+ 2589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:36:40.052241 (+ 4374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:36:40.052246 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:36:40.052249 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:36:40.052686 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:36:40.053176 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:36:40.053180 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:36:40.053182 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:36:40.053323 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:36:40.053983 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:36:40.053987 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:36:40.054113 (+ 126us) client_negotiation.cc:769] Sending connection context
0926 16:36:40.054242 (+ 129us) client_negotiation.cc:240] Negotiation successful
0926 16:36:40.054488 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 16:36:40.054906 29204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:40.039297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59344 (local address 127.6.33.61:35415)
0926 16:36:40.039517 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:40.039523 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:40.039547 (+ 24us) server_negotiation.cc:407] Connection header received
0926 16:36:40.039632 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:40.039637 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:40.039717 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:40.039831 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:40.040727 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:40.041867 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:40.043387 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:40.044074 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:40.044205 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:40.048102 (+ 3897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:40.048111 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:40.048118 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:36:40.048165 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:40.052072 (+ 3907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:40.052877 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:40.052883 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:40.052886 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:40.053034 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:40.053440 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:40.053444 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:40.053448 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:40.053823 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:40.053970 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:40.054383 (+ 413us) server_negotiation.cc:299] Negotiation successful
0926 16:36:40.054622 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 16:36:40.055953 29125 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.33.60" port: 39025 } has no permanent_uuid. Determining permanent_uuid...
I20260926 16:36:40.057144 29128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:40.056712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:39025 (local address 127.0.0.1:56106)
0926 16:36:40.056904 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:40.057001 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.60:39025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260926 16:36:40.058158 29128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:40.057836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:39025 (local address 127.0.0.1:56108)
0926 16:36:40.057976 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:40.058062 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.60:39025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 16:36:40.058377 29125 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.33.60:39025: Network error: Client connection negotiation failed: client connection to 127.6.33.60:39025: connect: Connection refused (error 111)
I20260926 16:36:40.109633 29125 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.33.60" port: 39025 } attempt: 1
I20260926 16:36:40.110795 29128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:40.110321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:39025 (local address 127.0.0.1:56116)
0926 16:36:40.110520 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:40.110620 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.60:39025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260926 16:36:40.111789 29128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:40.111475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:39025 (local address 127.0.0.1:56132)
0926 16:36:40.111623 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:40.111692 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.60:39025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 16:36:40.112031 29125 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.33.60:39025: Network error: Client connection negotiation failed: client connection to 127.6.33.60:39025: connect: Connection refused (error 111)
I20260926 16:36:40.138510 29199 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.33.60" port: 39025 } attempt: 3
I20260926 16:36:40.139693 29203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:40.139201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:39025 (local address 127.0.0.1:56144)
0926 16:36:40.139426 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:40.139529 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.60:39025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260926 16:36:40.141047 29203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:40.140490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:39025 (local address 127.0.0.1:56152)
0926 16:36:40.140748 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:40.140883 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.60:39025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146}
W20260926 16:36:40.141283 29199 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.33.60:39025: Network error: Client connection negotiation failed: client connection to 127.6.33.60:39025: connect: Connection refused (error 111)
I20260926 16:36:40.180275 29125 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.33.60" port: 39025 } attempt: 2
I20260926 16:36:40.181591 29128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:40.181087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:39025 (local address 127.0.0.1:56168)
0926 16:36:40.181331 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:40.181420 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.60:39025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260926 16:36:40.182615 29128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:40.182283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:39025 (local address 127.0.0.1:56184)
0926 16:36:40.182430 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:40.182513 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.60:39025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 16:36:40.182816 29125 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.33.60:39025: Network error: Client connection negotiation failed: client connection to 127.6.33.60:39025: connect: Connection refused (error 111)
W20260926 16:36:40.277616 29207 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 16:36:40.278014 29207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:36:40.278152 29207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:36:40.278295 29207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260926 16:36:40.284551 29199 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.33.60" port: 39025 } attempt: 4
W20260926 16:36:40.285934 29207 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
I20260926 16:36:40.285940 29203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:40.285322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:39025 (local address 127.0.0.1:56196)
0926 16:36:40.285586 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:40.285705 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.60:39025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260926 16:36:40.286064 29207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:36:40.286158 29207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:36:40.286263 29207 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:36:40.286345 29207 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:36:40.287009 29203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:40.286648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:39025 (local address 127.0.0.1:56202)
0926 16:36:40.286830 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:40.286908 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.60:39025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 16:36:40.287205 29199 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.33.60:39025: Network error: Client connection negotiation failed: client connection to 127.6.33.60:39025: connect: Connection refused (error 111)
I20260926 16:36:40.289006 29125 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.33.60" port: 39025 } attempt: 3
I20260926 16:36:40.290104 29128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:40.289674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:39025 (local address 127.0.0.1:56204)
0926 16:36:40.289859 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:40.289960 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.60:39025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260926 16:36:40.291069 29128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:40.290706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:39025 (local address 127.0.0.1:56220)
0926 16:36:40.290899 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:40.290975 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.33.60:39025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260926 16:36:40.291285 29125 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.33.60:39025: Network error: Client connection negotiation failed: client connection to 127.6.33.60:39025: connect: Connection refused (error 111)
I20260926 16:36:40.297546 29207 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.33.20:33859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.33.62:45999,127.6.33.61:35415,127.6.33.60:39025
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/krb5kdc/kudu_127.6.33.60.keytab
--principal=kudu/127.6.33.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.33.60:39025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.33.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:36:40.298926 29207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:36:40.300285 29207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:36:40.312160 29212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:36:40.312192 29213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:36:40.312124 29215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:36:40.313321 29207 server_base.cc:1034] running on GCE node
I20260926 16:36:40.314198 29207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:36:40.315686 29207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:36:40.316893 29207 hybrid_clock.cc:648] HybridClock initialized: now 1790440600316825 us; error 40 us; skew 500 ppm
I20260926 16:36:40.317277 29207 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:36:40 dist-test-slave-khd3 krb5kdc[29045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440600, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:36:40.320472 29207 init.cc:377] Logged in from keytab as kudu/127.6.33.60@KRBTEST.COM (short username kudu)
I20260926 16:36:40.321905 29207 webserver.cc:460] Webserver started at http://127.6.33.60:33639/ using document root <none> and password file <none>
I20260926 16:36:40.322546 29207 fs_manager.cc:359] Metadata directory not provided
I20260926 16:36:40.322628 29207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:36:40.322957 29207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:36:40.326063 29207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/minicluster-data/master-2/data/instance:
uuid: "7d940921bf5146c784532fb5c01148cc"
format_stamp: "Formatted at 2026-09-26 16:36:40 on dist-test-slave-khd3"
server_key: "40e019d7e961f203bd1928ddc30fc619"
server_key_iv: "ca6b24a40e3c04234097f476f7b680e0"
server_key_version: "encryptionkey@0"
I20260926 16:36:40.326848 29207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/minicluster-data/master-2/wal/instance:
uuid: "7d940921bf5146c784532fb5c01148cc"
format_stamp: "Formatted at 2026-09-26 16:36:40 on dist-test-slave-khd3"
server_key: "40e019d7e961f203bd1928ddc30fc619"
server_key_iv: "ca6b24a40e3c04234097f476f7b680e0"
server_key_version: "encryptionkey@0"
I20260926 16:36:40.332520 29207 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 16:36:40.336215 29222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:36:40.338047 29207 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:36:40.338230 29207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/minicluster-data/master-2/data,/tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/minicluster-data/master-2/wal
uuid: "7d940921bf5146c784532fb5c01148cc"
format_stamp: "Formatted at 2026-09-26 16:36:40 on dist-test-slave-khd3"
server_key: "40e019d7e961f203bd1928ddc30fc619"
server_key_iv: "ca6b24a40e3c04234097f476f7b680e0"
server_key_version: "encryptionkey@0"
I20260926 16:36:40.338482 29207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:36:40.363719 29207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:36:40.368432 29207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:36:40.368723 29207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:36:40.387907 29207 rpc_server.cc:307] RPC server started. Bound to: 127.6.33.60:39025
I20260926 16:36:40.387929 29273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.33.60:39025 every 8 connection(s)
I20260926 16:36:40.389425 29207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/minicluster-data/master-2/data/info.pb
I20260926 16:36:40.393625 29274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:36:40.399048 6276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxavWi/build/asan/bin/kudu as pid 29207
I20260926 16:36:40.399236 6276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxavWi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440233850748-6276-0/minicluster-data/master-2/wal/instance
I20260926 16:36:40.399761 6276 external_mini_cluster.cc:1428] Setting key 6aca33fdc34bd829973302f7e925ec33
I20260926 16:36:40.401239 29274 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.33.62" port: 45999 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 16:36:40 dist-test-slave-khd3 krb5kdc[29045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440600, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.60@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
Sep 26 16:36:40 dist-test-slave-khd3 krb5kdc[29045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.60@KRBTEST.COM
I20260926 16:36:40.428053 29279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:40.404264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55592 (local address 127.6.33.62:45999)
0926 16:36:40.405722 (+ 1458us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:40.405730 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:40.405750 (+ 20us) server_negotiation.cc:407] Connection header received
0926 16:36:40.405814 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:40.405817 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:40.405885 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:40.406006 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:40.407302 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:40.408022 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:40.409575 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:40.409965 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:40.410102 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:40.421845 (+ 11743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:40.421851 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:40.421854 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:36:40.421897 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:40.424903 (+ 3006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:40.425714 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:40.425719 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:40.425721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:40.425859 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:40.426301 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:40.426310 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:40.426312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:40.426579 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:40.426735 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:40.427443 (+ 708us) server_negotiation.cc:299] Negotiation successful
0926 16:36:40.427671 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1330,"thread_start_us":1223,"threads_started":1}
I20260926 16:36:40.429198 29277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:40.401848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56232 (local address 127.6.33.60:39025)
0926 16:36:40.402779 (+ 931us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:40.402791 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:40.402836 (+ 45us) server_negotiation.cc:407] Connection header received
0926 16:36:40.405204 (+ 2368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:40.405247 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:40.405683 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:40.405904 (+ 221us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:40.407442 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:40.408612 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:40.409963 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:40.410521 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:40.410672 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:40.422376 (+ 11704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:40.422385 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:40.422399 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:36:40.422427 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:40.425403 (+ 2976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:40.426099 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:40.426105 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:40.426107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:40.426245 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:40.426644 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:40.426648 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:40.426650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:40.427110 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:40.427433 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:40.427750 (+ 317us) server_negotiation.cc:299] Negotiation successful
0926 16:36:40.428227 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":701,"thread_start_us":447,"threads_started":1}
I20260926 16:36:40.429236 29278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:40.404208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.62:45999 (local address 127.0.0.1:55592)
0926 16:36:40.405042 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:40.405093 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 16:36:40.405443 (+ 350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:36:40.406078 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:36:40.406085 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:36:40.406729 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:36:40.407114 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:36:40.407132 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:36:40.408190 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:36:40.408195 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:36:40.409416 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:36:40.409427 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:36:40.410207 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:36:40.410217 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:36:40.410432 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:40.419060 (+ 8628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:36:40.419087 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:36:40.421593 (+ 2506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:36:40.425048 (+ 3455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:36:40.425068 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:36:40.425082 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:36:40.425571 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:36:40.425973 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:36:40.425979 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:36:40.425982 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:36:40.426181 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:36:40.426707 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:36:40.426715 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:36:40.427106 (+ 391us) client_negotiation.cc:769] Sending connection context
0926 16:36:40.427354 (+ 248us) client_negotiation.cc:240] Negotiation successful
0926 16:36:40.427842 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"mutex_wait_us":1,"thread_start_us":344,"threads_started":1}
I20260926 16:36:40.431286 29274 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.33.61" port: 35415 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 16:36:40 dist-test-slave-khd3 krb5kdc[29045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440600, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.60@KRBTEST.COM for kudu/127.6.33.61@KRBTEST.COM
Sep 26 16:36:40 dist-test-slave-khd3 krb5kdc[29045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.62@KRBTEST.COM
I20260926 16:36:40.446314 29278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:40.432056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.61:35415 (local address 127.0.0.1:59360)
0926 16:36:40.432784 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:40.432801 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 16:36:40.432914 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:36:40.433240 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:36:40.433245 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:36:40.433640 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:36:40.433779 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:36:40.433787 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:36:40.435050 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:36:40.435056 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:36:40.436157 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:36:40.436169 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:36:40.437118 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:36:40.437123 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:36:40.437325 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:40.437976 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:36:40.437998 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:36:40.439913 (+ 1915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:36:40.443450 (+ 3537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:36:40.443470 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:36:40.443477 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:36:40.444079 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:36:40.444480 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:36:40.444486 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:36:40.444491 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:36:40.444683 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:36:40.445316 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:36:40.445321 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:36:40.445469 (+ 148us) client_negotiation.cc:769] Sending connection context
0926 16:36:40.445656 (+ 187us) client_negotiation.cc:240] Negotiation successful
0926 16:36:40.445946 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606}
I20260926 16:36:40.446344 29204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:40.432060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59360 (local address 127.6.33.61:35415)
0926 16:36:40.432244 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:40.432249 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:40.432882 (+ 633us) server_negotiation.cc:407] Connection header received
0926 16:36:40.433012 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:40.433018 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:40.433105 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:40.433191 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:40.433934 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:40.434911 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:40.436294 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:40.436977 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:40.437099 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:40.440101 (+ 3002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:40.440108 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:40.440113 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:36:40.440154 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:40.443248 (+ 3094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:40.444208 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:40.444212 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:40.444214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:40.444362 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:40.444851 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:40.444854 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:40.444856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:40.445186 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:40.445357 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:40.445746 (+ 389us) server_negotiation.cc:299] Negotiation successful
0926 16:36:40.445952 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 16:36:40.447430 29274 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.33.60" port: 39025 } has no permanent_uuid. Determining permanent_uuid...
I20260926 16:36:40.448549 29279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:40.432598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55600 (local address 127.6.33.62:45999)
0926 16:36:40.432816 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:40.432823 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:40.432843 (+ 20us) server_negotiation.cc:407] Connection header received
0926 16:36:40.433131 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:40.433135 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:40.433228 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:40.433327 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:40.434625 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:40.435501 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:40.437098 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:40.437717 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:40.437852 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:40.441926 (+ 4074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:40.441931 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:40.441934 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:36:40.441964 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:40.445101 (+ 3137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:40.445740 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:40.445745 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:40.445747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:40.445857 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:40.446228 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:40.446233 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:40.446237 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:40.446772 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:40.447045 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:40.447591 (+ 546us) server_negotiation.cc:299] Negotiation successful
0926 16:36:40.447809 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Sep 26 16:36:40 dist-test-slave-khd3 krb5kdc[29045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440600, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.60@KRBTEST.COM for kudu/127.6.33.60@KRBTEST.COM
Sep 26 16:36:40 dist-test-slave-khd3 krb5kdc[29045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.33.61@KRBTEST.COM
I20260926 16:36:40.461230 29278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:40.448235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:39025 (local address 127.0.0.1:56248)
0926 16:36:40.448494 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:40.448508 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 16:36:40.448637 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:36:40.448998 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:36:40.449004 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:36:40.449415 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:36:40.449632 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:36:40.449645 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:36:40.450909 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:36:40.450913 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:36:40.451750 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:36:40.451760 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:36:40.452459 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:36:40.452465 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:36:40.452677 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:40.453367 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:36:40.453387 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:36:40.455544 (+ 2157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:36:40.458531 (+ 2987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:36:40.458537 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:36:40.458541 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:36:40.459094 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:36:40.459452 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:36:40.459459 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:36:40.459462 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:36:40.459624 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:36:40.460256 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:36:40.460264 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:36:40.460463 (+ 199us) client_negotiation.cc:769] Sending connection context
0926 16:36:40.460646 (+ 183us) client_negotiation.cc:240] Negotiation successful
0926 16:36:40.460939 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196}
I20260926 16:36:40.461535 29277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:40.448234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56248 (local address 127.6.33.60:39025)
0926 16:36:40.448416 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:40.448420 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:40.448576 (+ 156us) server_negotiation.cc:407] Connection header received
0926 16:36:40.448790 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:40.448794 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:40.448856 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:40.448933 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:40.449768 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:40.450795 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:40.451891 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:40.452316 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:40.452411 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:40.455727 (+ 3316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:40.455732 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:40.455735 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:36:40.455761 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:40.458351 (+ 2590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:40.459208 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:40.459212 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:40.459214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:40.459327 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:40.459734 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:40.459739 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:40.459742 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:40.460091 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:40.460244 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:40.460801 (+ 557us) server_negotiation.cc:299] Negotiation successful
0926 16:36:40.461128 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":55}
I20260926 16:36:40.465875 29204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:40.450294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59374 (local address 127.6.33.61:35415)
0926 16:36:40.450453 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:40.450456 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:40.450471 (+ 15us) server_negotiation.cc:407] Connection header received
0926 16:36:40.450591 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:40.450595 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:40.450648 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:40.450721 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:40.451941 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:40.452804 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:40.454174 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:40.454702 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:40.454796 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:40.459556 (+ 4760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:40.459562 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:40.459566 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:36:40.459598 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:40.463135 (+ 3537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:40.463795 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:40.463801 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:40.463805 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:40.463953 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:40.464324 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:40.464330 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:40.464333 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:40.464610 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:40.464915 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:40.465217 (+ 302us) server_negotiation.cc:299] Negotiation successful
0926 16:36:40.465429 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 16:36:40.466596 29274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d940921bf5146c784532fb5c01148cc: Bootstrap starting.
I20260926 16:36:40.468487 29125 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.33.60" port: 39025 } attempt: 4
I20260926 16:36:40.470748 29274 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d940921bf5146c784532fb5c01148cc: Neither blocks nor log segments found. Creating new log.
I20260926 16:36:40.471153 6276 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:36:40.471817 29274 log.cc:824] T 00000000000000000000000000000000 P 7d940921bf5146c784532fb5c01148cc: Log is configured to *not* fsync() on all Append() calls
I20260926 16:36:40.475365 29274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d940921bf5146c784532fb5c01148cc: No bootstrap required, opened a new log
I20260926 16:36:40.473698 29100 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "3b64decc87f94872a607612ef8fc2a33"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55600
Sep 26 16:36:40 dist-test-slave-khd3 krb5kdc[29045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440599, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.62@KRBTEST.COM for kudu/127.6.33.60@KRBTEST.COM
I20260926 16:36:40.485165 29274 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d940921bf5146c784532fb5c01148cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b64decc87f94872a607612ef8fc2a33" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 45999 } } peers { permanent_uuid: "4255e94899314359b96868d43dea1017" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 35415 } } peers { permanent_uuid: "7d940921bf5146c784532fb5c01148cc" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 39025 } }
I20260926 16:36:40.485648 29274 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d940921bf5146c784532fb5c01148cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d940921bf5146c784532fb5c01148cc, State: Initialized, Role: FOLLOWER
I20260926 16:36:40.486141 29274 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d940921bf5146c784532fb5c01148cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b64decc87f94872a607612ef8fc2a33" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 45999 } } peers { permanent_uuid: "4255e94899314359b96868d43dea1017" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 35415 } } peers { permanent_uuid: "7d940921bf5146c784532fb5c01148cc" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 39025 } }
I20260926 16:36:40.487222 29281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d940921bf5146c784532fb5c01148cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b64decc87f94872a607612ef8fc2a33" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 45999 } } peers { permanent_uuid: "4255e94899314359b96868d43dea1017" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 35415 } } peers { permanent_uuid: "7d940921bf5146c784532fb5c01148cc" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 39025 } } }
I20260926 16:36:40.487648 29281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d940921bf5146c784532fb5c01148cc [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:36:40.488631 29274 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d940921bf5146c784532fb5c01148cc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:36:40.488706 29128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:40.469194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:39025 (local address 127.0.0.1:56264)
0926 16:36:40.469434 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:40.469453 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 16:36:40.469557 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:36:40.472811 (+ 3254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:36:40.472818 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:36:40.473344 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:36:40.473586 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:36:40.473612 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:36:40.474974 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:36:40.474981 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:36:40.476529 (+ 1548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:36:40.476544 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:36:40.477660 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:36:40.477667 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:36:40.477878 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:40.478579 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:36:40.478604 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:36:40.481508 (+ 2904us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:36:40.485640 (+ 4132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:36:40.485649 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:36:40.485652 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:36:40.486098 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:36:40.486532 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:36:40.486536 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:36:40.486540 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:36:40.486766 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:36:40.487626 (+ 860us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:36:40.487635 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:36:40.487810 (+ 175us) client_negotiation.cc:769] Sending connection context
0926 16:36:40.487986 (+ 176us) client_negotiation.cc:240] Negotiation successful
0926 16:36:40.488339 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132}
I20260926 16:36:40.488766 29277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:40.469273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56264 (local address 127.6.33.60:39025)
0926 16:36:40.472455 (+ 3182us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:40.472463 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:40.472483 (+ 20us) server_negotiation.cc:407] Connection header received
0926 16:36:40.472547 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:40.472562 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:40.472640 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:40.472750 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:40.473775 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:40.474806 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:40.476685 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:40.477497 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:40.477642 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:40.481709 (+ 4067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:40.481716 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:40.481721 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:36:40.481757 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:40.485480 (+ 3723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:40.486222 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:40.486227 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:40.486231 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:40.486393 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:40.486899 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:40.486905 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:40.486914 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:40.487369 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:40.487555 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:40.488133 (+ 578us) server_negotiation.cc:299] Negotiation successful
0926 16:36:40.488364 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3019}
I20260926 16:36:40.495348 29125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b64decc87f94872a607612ef8fc2a33: Bootstrap starting.
I20260926 16:36:40.499313 29125 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b64decc87f94872a607612ef8fc2a33: Neither blocks nor log segments found. Creating new log.
I20260926 16:36:40.500669 29125 log.cc:824] T 00000000000000000000000000000000 P 3b64decc87f94872a607612ef8fc2a33: Log is configured to *not* fsync() on all Append() calls
I20260926 16:36:40.504060 29125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b64decc87f94872a607612ef8fc2a33: No bootstrap required, opened a new log
W20260926 16:36:40.509222 29292 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7d940921bf5146c784532fb5c01148cc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 16:36:40.509547 29292 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 16:36:40.495994 29207 master_runner.cc:186] Error getting master registration for 127.6.33.62:45999: OK, instance_id { permanent_uuid: "3b64decc87f94872a607612ef8fc2a33" instance_seqno: 1790440599322473 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 16:36:40.510028 29207 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 16:36:40.511310 29125 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b64decc87f94872a607612ef8fc2a33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b64decc87f94872a607612ef8fc2a33" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 45999 } } peers { permanent_uuid: "4255e94899314359b96868d43dea1017" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 35415 } } peers { permanent_uuid: "7d940921bf5146c784532fb5c01148cc" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 39025 } }
I20260926 16:36:40.511667 29125 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b64decc87f94872a607612ef8fc2a33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b64decc87f94872a607612ef8fc2a33, State: Initialized, Role: FOLLOWER
I20260926 16:36:40.512159 29125 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b64decc87f94872a607612ef8fc2a33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b64decc87f94872a607612ef8fc2a33" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 45999 } } peers { permanent_uuid: "4255e94899314359b96868d43dea1017" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 35415 } } peers { permanent_uuid: "7d940921bf5146c784532fb5c01148cc" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 39025 } }
I20260926 16:36:40.513186 29295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b64decc87f94872a607612ef8fc2a33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b64decc87f94872a607612ef8fc2a33" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 45999 } } peers { permanent_uuid: "4255e94899314359b96868d43dea1017" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 35415 } } peers { permanent_uuid: "7d940921bf5146c784532fb5c01148cc" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 39025 } } }
I20260926 16:36:40.513657 29295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b64decc87f94872a607612ef8fc2a33 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:36:40.514385 29125 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b64decc87f94872a607612ef8fc2a33 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:36:40.514523 29100 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b64decc87f94872a607612ef8fc2a33 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 16:36:40.514719 29100 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b64decc87f94872a607612ef8fc2a33 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:36:40.519012 29100 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b64decc87f94872a607612ef8fc2a33 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b64decc87f94872a607612ef8fc2a33" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 45999 } } peers { permanent_uuid: "4255e94899314359b96868d43dea1017" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 35415 } } peers { permanent_uuid: "7d940921bf5146c784532fb5c01148cc" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 39025 } }
I20260926 16:36:40.521091 29100 leader_election.cc:288] T 00000000000000000000000000000000 P 3b64decc87f94872a607612ef8fc2a33 [CANDIDATE]: Term 1 election: Requested vote from peers 4255e94899314359b96868d43dea1017 (127.6.33.61:35415), 7d940921bf5146c784532fb5c01148cc (127.6.33.60:39025)
I20260926 16:36:40.522398 29249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3b64decc87f94872a607612ef8fc2a33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7d940921bf5146c784532fb5c01148cc"
I20260926 16:36:40.522930 29249 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d940921bf5146c784532fb5c01148cc [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:36:40.527333 29249 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7d940921bf5146c784532fb5c01148cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b64decc87f94872a607612ef8fc2a33 in term 1.
I20260926 16:36:40.528298 29076 leader_election.cc:302] T 00000000000000000000000000000000 P 3b64decc87f94872a607612ef8fc2a33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b64decc87f94872a607612ef8fc2a33, 7d940921bf5146c784532fb5c01148cc; no voters:
I20260926 16:36:40.529014 29295 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b64decc87f94872a607612ef8fc2a33 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:36:40.533610 29295 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b64decc87f94872a607612ef8fc2a33 [term 1 LEADER]: Becoming Leader. State: Replica: 3b64decc87f94872a607612ef8fc2a33, State: Running, Role: LEADER
I20260926 16:36:40.534401 29295 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b64decc87f94872a607612ef8fc2a33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b64decc87f94872a607612ef8fc2a33" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 45999 } } peers { permanent_uuid: "4255e94899314359b96868d43dea1017" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 35415 } } peers { permanent_uuid: "7d940921bf5146c784532fb5c01148cc" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 39025 } }
I20260926 16:36:40.540582 29300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b64decc87f94872a607612ef8fc2a33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b64decc87f94872a607612ef8fc2a33. Latest consensus state: current_term: 1 leader_uuid: "3b64decc87f94872a607612ef8fc2a33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b64decc87f94872a607612ef8fc2a33" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 45999 } } peers { permanent_uuid: "4255e94899314359b96868d43dea1017" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 35415 } } peers { permanent_uuid: "7d940921bf5146c784532fb5c01148cc" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 39025 } } }
I20260926 16:36:40.541380 29300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b64decc87f94872a607612ef8fc2a33 [sys.catalog]: This master's current role is: LEADER
I20260926 16:36:40.546792 29304 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:36:40.561385 29304 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260926 16:36:40.561710 29079 tablet.cc:2369] T 00000000000000000000000000000000 P 3b64decc87f94872a607612ef8fc2a33: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:36:40.586411 29199 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.33.60" port: 39025 } attempt: 5
I20260926 16:36:40.591528 29318 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 16:36:40.596359 29058 master_runner.cc:186] Error getting master registration for 127.6.33.61:35415: OK, instance_id { permanent_uuid: "4255e94899314359b96868d43dea1017" instance_seqno: 1790440599858489 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 16:36:40.608340 29058 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 16:36:40.621016 29228 tablet.cc:2369] T 00000000000000000000000000000000 P 7d940921bf5146c784532fb5c01148cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:36:40.629446 29249 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7d940921bf5146c784532fb5c01148cc [term 1 FOLLOWER]: Refusing update from remote peer 3b64decc87f94872a607612ef8fc2a33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:36:40.630609 29300 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3b64decc87f94872a607612ef8fc2a33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d940921bf5146c784532fb5c01148cc" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 39025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Sep 26 16:36:40 dist-test-slave-khd3 krb5kdc[29045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440599, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.33.61@KRBTEST.COM for kudu/127.6.33.60@KRBTEST.COM
I20260926 16:36:40.657187 29281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d940921bf5146c784532fb5c01148cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b64decc87f94872a607612ef8fc2a33. Latest consensus state: current_term: 1 leader_uuid: "3b64decc87f94872a607612ef8fc2a33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b64decc87f94872a607612ef8fc2a33" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 45999 } } peers { permanent_uuid: "4255e94899314359b96868d43dea1017" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 35415 } } peers { permanent_uuid: "7d940921bf5146c784532fb5c01148cc" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 39025 } } }
I20260926 16:36:40.657568 29281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d940921bf5146c784532fb5c01148cc [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:36:40.658124 29203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:40.587341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.33.60:39025 (local address 127.0.0.1:56288)
0926 16:36:40.587711 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:40.587738 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 16:36:40.587890 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:36:40.612569 (+ 24679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:36:40.612576 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:36:40.613208 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:36:40.613438 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:36:40.613452 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:36:40.628212 (+ 14760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:36:40.628224 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:36:40.629753 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:36:40.629774 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:36:40.634736 (+ 4962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:36:40.634741 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:36:40.634958 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:40.635684 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:36:40.635712 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:36:40.639694 (+ 3982us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:36:40.649282 (+ 9588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:36:40.649289 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:36:40.649294 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:36:40.649905 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:36:40.652318 (+ 2413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:36:40.652326 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:36:40.652331 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:36:40.652555 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:36:40.656888 (+ 4333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:36:40.656903 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:36:40.657138 (+ 235us) client_negotiation.cc:769] Sending connection context
0926 16:36:40.657297 (+ 159us) client_negotiation.cc:240] Negotiation successful
0926 16:36:40.657674 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191}
I20260926 16:36:40.662516 29319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:40.588290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56288 (local address 127.6.33.60:39025)
0926 16:36:40.612134 (+ 23844us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:40.612144 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:40.612179 (+ 35us) server_negotiation.cc:407] Connection header received
0926 16:36:40.612253 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:40.612259 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:40.612366 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:40.612527 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:40.620963 (+ 8436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:40.622856 (+ 1893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:40.630022 (+ 7166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:40.634544 (+ 4522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:40.634702 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:40.639999 (+ 5297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:40.640011 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:40.640016 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:36:40.640065 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:40.649076 (+ 9011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:40.650086 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:40.650092 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:40.650095 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:40.650289 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:40.652725 (+ 2436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:40.652730 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:40.652734 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:40.653173 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:40.661287 (+ 8114us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:40.661448 (+ 161us) server_negotiation.cc:299] Negotiation successful
0926 16:36:40.661745 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23649,"thread_start_us":24905,"threads_started":1}
I20260926 16:36:40.666145 29300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b64decc87f94872a607612ef8fc2a33 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3b64decc87f94872a607612ef8fc2a33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b64decc87f94872a607612ef8fc2a33" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 45999 } } peers { permanent_uuid: "4255e94899314359b96868d43dea1017" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 35415 } } peers { permanent_uuid: "7d940921bf5146c784532fb5c01148cc" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 39025 } } }
I20260926 16:36:40.666463 29300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b64decc87f94872a607612ef8fc2a33 [sys.catalog]: This master's current role is: LEADER
I20260926 16:36:40.671687 29279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:40.569060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55602 (local address 127.6.33.62:45999)
0926 16:36:40.569494 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:40.569502 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:40.617821 (+ 48319us) server_negotiation.cc:407] Connection header received
0926 16:36:40.618154 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:40.618160 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:40.618249 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:40.618359 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:40.637409 (+ 19050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:40.638643 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:40.646203 (+ 7560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:40.646933 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:40.647047 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:40.660024 (+ 12977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:40.660036 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:40.660042 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:36:40.660128 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:40.664106 (+ 3978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:40.667894 (+ 3788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:40.667899 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:40.667903 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:40.668115 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:40.669102 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:40.669109 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:40.669118 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:40.669540 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:40.669814 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:40.671013 (+ 1199us) server_negotiation.cc:299] Negotiation successful
0926 16:36:40.671291 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291}
I20260926 16:36:40.675987 29199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4255e94899314359b96868d43dea1017: Bootstrap starting.
I20260926 16:36:40.681734 29204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:40.568845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59380 (local address 127.6.33.61:35415)
0926 16:36:40.569318 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:40.569327 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:40.611248 (+ 41921us) server_negotiation.cc:407] Connection header received
0926 16:36:40.611501 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:40.611507 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:40.611612 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:40.611744 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:40.633469 (+ 21725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:40.634570 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:40.650462 (+ 15892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:40.651159 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:40.651262 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:40.668312 (+ 17050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:40.668321 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:40.668336 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:36:40.668382 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:40.672564 (+ 4182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:40.676549 (+ 3985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:40.676556 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:40.676560 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:40.676905 (+ 345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:40.677892 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:40.677899 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:40.677903 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:40.678801 (+ 898us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:40.679265 (+ 464us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:40.680495 (+ 1230us) server_negotiation.cc:299] Negotiation successful
0926 16:36:40.680999 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285}
I20260926 16:36:40.685052 29304 catalog_manager.cc:1347] Generated new cluster ID: d61f82b27762422fba5a7bd8e66f33fa
I20260926 16:36:40.685176 29304 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:36:40.689687 29199 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4255e94899314359b96868d43dea1017: Neither blocks nor log segments found. Creating new log.
I20260926 16:36:40.692701 29199 log.cc:824] T 00000000000000000000000000000000 P 4255e94899314359b96868d43dea1017: Log is configured to *not* fsync() on all Append() calls
I20260926 16:36:40.697340 29277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:40.575593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56272 (local address 127.6.33.60:39025)
0926 16:36:40.575845 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:40.575855 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:40.633291 (+ 57436us) server_negotiation.cc:407] Connection header received
0926 16:36:40.646318 (+ 13027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:40.646325 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:40.646413 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:40.646560 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:40.653359 (+ 6799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:40.654497 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:40.663214 (+ 8717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:40.674489 (+ 11275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:40.674652 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:40.684958 (+ 10306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:40.684969 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:40.684977 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:36:40.685038 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:40.689170 (+ 4132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:40.691288 (+ 2118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:40.691301 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:40.691307 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:40.691814 (+ 507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:40.692901 (+ 1087us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:40.692941 (+ 40us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:40.692946 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:40.693999 (+ 1053us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:40.694631 (+ 632us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:40.695988 (+ 1357us) server_negotiation.cc:299] Negotiation successful
0926 16:36:40.696524 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 16:36:40.698064 29281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d940921bf5146c784532fb5c01148cc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3b64decc87f94872a607612ef8fc2a33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b64decc87f94872a607612ef8fc2a33" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 45999 } } peers { permanent_uuid: "4255e94899314359b96868d43dea1017" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 35415 } } peers { permanent_uuid: "7d940921bf5146c784532fb5c01148cc" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 39025 } } }
I20260926 16:36:40.698347 29281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d940921bf5146c784532fb5c01148cc [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:36:40.707147 29304 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:36:40.709281 29304 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:36:40.713404 29199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4255e94899314359b96868d43dea1017: No bootstrap required, opened a new log
I20260926 16:36:40.720858 29304 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b64decc87f94872a607612ef8fc2a33: Generated new TSK 0
I20260926 16:36:40.722326 29304 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:36:40.734030 29199 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4255e94899314359b96868d43dea1017 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b64decc87f94872a607612ef8fc2a33" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 45999 } } peers { permanent_uuid: "4255e94899314359b96868d43dea1017" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 35415 } } peers { permanent_uuid: "7d940921bf5146c784532fb5c01148cc" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 39025 } }
I20260926 16:36:40.734670 29199 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4255e94899314359b96868d43dea1017 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4255e94899314359b96868d43dea1017, State: Initialized, Role: FOLLOWER
I20260926 16:36:40.735343 29199 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4255e94899314359b96868d43dea1017 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b64decc87f94872a607612ef8fc2a33" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 45999 } } peers { permanent_uuid: "4255e94899314359b96868d43dea1017" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 35415 } } peers { permanent_uuid: "7d940921bf5146c784532fb5c01148cc" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 39025 } }
I20260926 16:36:40.753403 29325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4255e94899314359b96868d43dea1017 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b64decc87f94872a607612ef8fc2a33" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 45999 } } peers { permanent_uuid: "4255e94899314359b96868d43dea1017" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 35415 } } peers { permanent_uuid: "7d940921bf5146c784532fb5c01148cc" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 39025 } } }
I20260926 16:36:40.754022 29325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4255e94899314359b96868d43dea1017 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:36:40.755000 29199 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4255e94899314359b96868d43dea1017 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:36:40.755661 29173 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4255e94899314359b96868d43dea1017 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:36:40.761643 29173 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4255e94899314359b96868d43dea1017 [term 1 FOLLOWER]: Refusing update from remote peer 3b64decc87f94872a607612ef8fc2a33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:36:40.755924 29174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3b64decc87f94872a607612ef8fc2a33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4255e94899314359b96868d43dea1017"
I20260926 16:36:40.767473 29174 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4255e94899314359b96868d43dea1017 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b64decc87f94872a607612ef8fc2a33 in term 1.
I20260926 16:36:40.777799 29300 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3b64decc87f94872a607612ef8fc2a33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4255e94899314359b96868d43dea1017" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 35415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:36:40.828617 29334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4255e94899314359b96868d43dea1017 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3b64decc87f94872a607612ef8fc2a33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b64decc87f94872a607612ef8fc2a33" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 45999 } } peers { permanent_uuid: "4255e94899314359b96868d43dea1017" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 35415 } } peers { permanent_uuid: "7d940921bf5146c784532fb5c01148cc" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 39025 } } }
I20260926 16:36:40.829121 29334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4255e94899314359b96868d43dea1017 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:36:40.829510 29325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4255e94899314359b96868d43dea1017 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b64decc87f94872a607612ef8fc2a33. Latest consensus state: current_term: 1 leader_uuid: "3b64decc87f94872a607612ef8fc2a33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b64decc87f94872a607612ef8fc2a33" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 45999 } } peers { permanent_uuid: "4255e94899314359b96868d43dea1017" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 35415 } } peers { permanent_uuid: "7d940921bf5146c784532fb5c01148cc" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 39025 } } }
I20260926 16:36:40.829802 29325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4255e94899314359b96868d43dea1017 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:36:40.859969 29342 catalog_manager.cc:1259] Loaded cluster ID: d61f82b27762422fba5a7bd8e66f33fa
I20260926 16:36:40.860106 29342 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4255e94899314359b96868d43dea1017: loading cluster ID for follower catalog manager: success
W20260926 16:36:40.867101 6276 mini_kdc.cc:348] Time spent kdestroy: real 0.129s user 0.004s sys 0.022s
I20260926 16:36:40.878911 29342 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4255e94899314359b96868d43dea1017: acquiring CA information for follower catalog manager: success
I20260926 16:36:40.882613 29342 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4255e94899314359b96868d43dea1017: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 16:36:40.919652 29204 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:36:40.905536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59382 (local address 127.6.33.61:35415)
0926 16:36:40.905713 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:40.905718 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:40.917739 (+ 12021us) server_negotiation.cc:407] Connection header received
0926 16:36:40.918076 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:40.918085 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:40.918172 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:40.918242 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:40.919390 (+ 1148us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59382: BlockingRecv error: recv got EOF from 127.0.0.1:59382 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260926 16:36:40.919648 29279 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:36:40.898721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55604 (local address 127.6.33.62:45999)
0926 16:36:40.898937 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:40.898943 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:40.917663 (+ 18720us) server_negotiation.cc:407] Connection header received
0926 16:36:40.917897 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:40.917906 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:40.918023 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:40.918128 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:40.919391 (+ 1263us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55604: BlockingRecv error: recv got EOF from 127.0.0.1:55604 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260926 16:36:40.922228 29277 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:36:40.905072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56300 (local address 127.6.33.60:39025)
0926 16:36:40.905253 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:40.905260 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:40.920344 (+ 15084us) server_negotiation.cc:407] Connection header received
0926 16:36:40.920612 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:40.920619 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:40.920711 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:40.920834 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:40.921886 (+ 1052us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56300: BlockingRecv error: recv got EOF from 127.0.0.1:56300 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 16:36:40.985077 29204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:40.974430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59388 (local address 127.6.33.61:35415)
0926 16:36:40.974627 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:40.974633 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:40.977366 (+ 2733us) server_negotiation.cc:407] Connection header received
0926 16:36:40.977724 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:40.977730 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:40.977971 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:40.978113 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:36:40.978797 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:40.979893 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:40.981963 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:40.982626 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:40.982728 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:40.983474 (+ 746us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:36:40.984137 (+ 663us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:36:40.984263 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:40.984530 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 16:36:40.984636 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260926 16:36:41.048269 29277 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:36:41.034874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56308 (local address 127.6.33.60:39025)
0926 16:36:41.035137 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:41.035143 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:41.046078 (+ 10935us) server_negotiation.cc:407] Connection header received
0926 16:36:41.046671 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:41.046677 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:41.046803 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:41.046887 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:41.048033 (+ 1146us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56308: BlockingRecv error: recv got EOF from 127.0.0.1:56308 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
I20260926 16:36:41.105993 29204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:41.086381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59404 (local address 127.6.33.61:35415)
0926 16:36:41.086627 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:41.086636 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:41.097978 (+ 11342us) server_negotiation.cc:407] Connection header received
0926 16:36:41.098589 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:41.098597 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:41.098699 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:41.098769 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:36:41.099549 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:41.100698 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:41.103218 (+ 2520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:41.103953 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:41.104075 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:41.104909 (+ 834us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:36:41.105098 (+ 189us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:36:41.105178 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:41.105524 (+ 346us) server_negotiation.cc:299] Negotiation successful
0926 16:36:41.105629 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260926 16:36:41.151017 29277 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:36:41.141046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56310 (local address 127.6.33.60:39025)
0926 16:36:41.141265 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:41.141272 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:41.148930 (+ 7658us) server_negotiation.cc:407] Connection header received
0926 16:36:41.149463 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:41.149470 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:41.149609 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:41.149701 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:41.150754 (+ 1053us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56310: BlockingRecv error: recv got EOF from 127.0.0.1:56310 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
I20260926 16:36:41.206544 29204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:41.192329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59408 (local address 127.6.33.61:35415)
0926 16:36:41.192556 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:41.192563 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:41.199411 (+ 6848us) server_negotiation.cc:407] Connection header received
0926 16:36:41.199777 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:41.199784 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:41.199886 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:41.199966 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:36:41.200703 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:41.201909 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:41.204153 (+ 2244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:41.204812 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:41.204912 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:41.205539 (+ 627us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:36:41.205705 (+ 166us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:36:41.205776 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:41.206049 (+ 273us) server_negotiation.cc:299] Negotiation successful
0926 16:36:41.206142 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260926 16:36:41.251053 29277 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:36:41.241427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56318 (local address 127.6.33.60:39025)
0926 16:36:41.241670 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:41.241679 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:41.249052 (+ 7373us) server_negotiation.cc:407] Connection header received
0926 16:36:41.249562 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:41.249568 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:41.249664 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:41.249738 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:41.250777 (+ 1039us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56318: BlockingRecv error: recv got EOF from 127.0.0.1:56318 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 16:36:41.306772 29204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:41.291300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59412 (local address 127.6.33.61:35415)
0926 16:36:41.291726 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:41.291737 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:41.300471 (+ 8734us) server_negotiation.cc:407] Connection header received
0926 16:36:41.300697 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:41.300703 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:41.300831 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:41.300940 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:36:41.301500 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:41.302679 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:41.304714 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:41.305261 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:41.305373 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:41.306014 (+ 641us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:36:41.306128 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:36:41.306208 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:41.306395 (+ 187us) server_negotiation.cc:299] Negotiation successful
0926 16:36:41.306469 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241}
W20260926 16:36:41.315732 29277 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:36:41.313026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56328 (local address 127.6.33.60:39025)
0926 16:36:41.313309 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:41.313318 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:41.313954 (+ 636us) server_negotiation.cc:407] Connection header received
0926 16:36:41.314260 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:41.314268 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:41.314354 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:41.314479 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:41.315412 (+ 933us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56328: BlockingRecv error: recv got EOF from 127.0.0.1:56328 (error 108)
Metrics: {"server-negotiator.queue_time_us":134}
I20260926 16:36:41.334731 29204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:41.327988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59422 (local address 127.6.33.61:35415)
0926 16:36:41.328248 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:41.328257 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:41.328921 (+ 664us) server_negotiation.cc:407] Connection header received
0926 16:36:41.329412 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:41.329419 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:41.329661 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:41.329764 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:36:41.330348 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:41.331111 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:41.332507 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:41.333103 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:41.333240 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:41.333637 (+ 397us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:36:41.333783 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:36:41.333873 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:41.334126 (+ 253us) server_negotiation.cc:299] Negotiation successful
0926 16:36:41.334245 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
W20260926 16:36:41.343954 29277 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:36:41.342198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56342 (local address 127.6.33.60:39025)
0926 16:36:41.342397 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:41.342404 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:41.342424 (+ 20us) server_negotiation.cc:407] Connection header received
0926 16:36:41.342703 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:41.342707 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:41.342780 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:41.342894 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:41.343731 (+ 837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56342: BlockingRecv error: recv got EOF from 127.0.0.1:56342 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 16:36:41.369918 29204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:41.363601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59430 (local address 127.6.33.61:35415)
0926 16:36:41.363839 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:41.363846 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:41.364233 (+ 387us) server_negotiation.cc:407] Connection header received
0926 16:36:41.364416 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:41.364422 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:41.364537 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:41.364666 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:36:41.365136 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:41.366315 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:41.367778 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:41.368438 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:41.368595 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:41.369113 (+ 518us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:36:41.369263 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:36:41.369366 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:41.369514 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 16:36:41.369628 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
W20260926 16:36:41.378841 29277 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:36:41.376577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56352 (local address 127.6.33.60:39025)
0926 16:36:41.376861 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:41.376866 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:41.377116 (+ 250us) server_negotiation.cc:407] Connection header received
0926 16:36:41.377357 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:41.377362 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:41.377438 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:41.377551 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:41.378548 (+ 997us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56352: BlockingRecv error: recv got EOF from 127.0.0.1:56352 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 16:36:41.380828 29397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b64decc87f94872a607612ef8fc2a33 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3b64decc87f94872a607612ef8fc2a33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b64decc87f94872a607612ef8fc2a33" member_type: VOTER last_known_addr { host: "127.6.33.62" port: 45999 } } peers { permanent_uuid: "4255e94899314359b96868d43dea1017" member_type: VOTER last_known_addr { host: "127.6.33.61" port: 35415 } } peers { permanent_uuid: "7d940921bf5146c784532fb5c01148cc" member_type: VOTER last_known_addr { host: "127.6.33.60" port: 39025 } } }
I20260926 16:36:41.381150 29397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b64decc87f94872a607612ef8fc2a33 [sys.catalog]: This master's current role is: LEADER
I20260926 16:36:41.420532 29204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:41.414589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59432 (local address 127.6.33.61:35415)
0926 16:36:41.414809 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:41.414819 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:41.415453 (+ 634us) server_negotiation.cc:407] Connection header received
0926 16:36:41.415759 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:41.415764 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:41.415836 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:41.415951 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:36:41.416414 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:41.417223 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:41.418544 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:41.419019 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:41.419160 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:41.419575 (+ 415us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:36:41.419685 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:36:41.419790 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:41.419975 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 16:36:41.420056 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260926 16:36:41.428712 29277 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:36:41.426487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56358 (local address 127.6.33.60:39025)
0926 16:36:41.426721 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:41.426727 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:41.427031 (+ 304us) server_negotiation.cc:407] Connection header received
0926 16:36:41.427281 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:41.427286 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:41.427374 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:41.427467 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:41.428466 (+ 999us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56358: BlockingRecv error: recv got EOF from 127.0.0.1:56358 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
I20260926 16:36:41.501296 29204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:41.496222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59444 (local address 127.6.33.61:35415)
0926 16:36:41.496500 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:41.496508 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:41.496721 (+ 213us) server_negotiation.cc:407] Connection header received
0926 16:36:41.496932 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:41.496937 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:41.497032 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:41.497119 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:36:41.497532 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:41.498356 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:41.499674 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:41.500101 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:41.500195 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:41.500567 (+ 372us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:36:41.500675 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:36:41.500745 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:41.500888 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 16:36:41.500950 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260926 16:36:41.509325 29277 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:36:41.507126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56374 (local address 127.6.33.60:39025)
0926 16:36:41.507478 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:41.507485 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:41.507775 (+ 290us) server_negotiation.cc:407] Connection header received
0926 16:36:41.507992 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:41.507996 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:41.508067 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:41.508175 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:41.509053 (+ 878us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56374: BlockingRecv error: recv got EOF from 127.0.0.1:56374 (error 108)
Metrics: {"server-negotiator.queue_time_us":179}
I20260926 16:36:41.511493 29292 catalog_manager.cc:1259] Loaded cluster ID: d61f82b27762422fba5a7bd8e66f33fa
I20260926 16:36:41.511572 29292 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7d940921bf5146c784532fb5c01148cc: loading cluster ID for follower catalog manager: success
I20260926 16:36:41.514369 29292 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7d940921bf5146c784532fb5c01148cc: acquiring CA information for follower catalog manager: success
I20260926 16:36:41.516233 29292 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7d940921bf5146c784532fb5c01148cc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 16:36:41.647809 29204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:41.642500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59448 (local address 127.6.33.61:35415)
0926 16:36:41.642683 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:41.642687 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:41.643045 (+ 358us) server_negotiation.cc:407] Connection header received
0926 16:36:41.643196 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:41.643200 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:41.643316 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:41.643409 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:36:41.643902 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:41.644611 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:41.646019 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:41.646467 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:41.646640 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:41.647072 (+ 432us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:36:41.647184 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:36:41.647284 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:41.647470 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 16:36:41.647553 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 16:36:41.660550 29277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:41.654202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56382 (local address 127.6.33.60:39025)
0926 16:36:41.654473 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:41.654479 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:41.654742 (+ 263us) server_negotiation.cc:407] Connection header received
0926 16:36:41.654900 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:41.654904 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:41.655103 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:41.655272 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:36:41.655726 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:41.656603 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:41.658071 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:41.658499 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:41.658621 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:41.659244 (+ 623us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:36:41.659847 (+ 603us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:36:41.659968 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:41.660110 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 16:36:41.660203 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 16:36:41.676561 29277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:41.666880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56398 (local address 127.6.33.60:39025)
0926 16:36:41.667085 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:41.667092 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:41.670375 (+ 3283us) server_negotiation.cc:407] Connection header received
0926 16:36:41.670605 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:41.670611 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:41.670708 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:41.670813 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:36:41.671496 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:41.672491 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:41.674312 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:41.675005 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:41.675149 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:41.675692 (+ 543us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:36:41.675856 (+ 164us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:36:41.675936 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:41.676165 (+ 229us) server_negotiation.cc:299] Negotiation successful
0926 16:36:41.676255 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 16:36:41.677285 29451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:41.666894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55610 (local address 127.6.33.62:45999)
0926 16:36:41.669037 (+ 2143us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:41.669046 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:41.669638 (+ 592us) server_negotiation.cc:407] Connection header received
0926 16:36:41.669907 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:41.669915 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:41.670179 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:41.670383 (+ 204us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:36:41.671026 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:41.672204 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:41.674713 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:41.675426 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:41.675573 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:41.676074 (+ 501us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:36:41.676571 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:36:41.676692 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:41.676861 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 16:36:41.676967 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1990,"thread_start_us":1880,"threads_started":1}
I20260926 16:36:41.677340 29204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:41.667163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59456 (local address 127.6.33.61:35415)
0926 16:36:41.667398 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:41.667407 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:41.669934 (+ 2527us) server_negotiation.cc:407] Connection header received
0926 16:36:41.670298 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:41.670304 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:41.670426 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:41.670517 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:36:41.671416 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:41.672153 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:41.674864 (+ 2711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:41.675524 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:41.675644 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:41.676294 (+ 650us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:36:41.676418 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:36:41.676486 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:41.676983 (+ 497us) server_negotiation.cc:299] Negotiation successful
0926 16:36:41.677075 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 16:36:41.682628 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 29058
I20260926 16:36:41.683673 29120 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:36:41.797130 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 29058
I20260926 16:36:41.820328 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 29132
I20260926 16:36:41.821568 29194 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:36:41.932487 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 29132
I20260926 16:36:41.952816 6276 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVxavWi/build/asan/bin/kudu pid 29207
I20260926 16:36:41.954188 29269 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:36:42.063822 6276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxavWi/build/asan/bin/kudu with pid 29207
2026-09-26T16:36:42Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4581 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9593 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (368179 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST